Glare is always present in optical acquisition systems, such as photocameras or the eye-bulb. As a consequence, images captured by sensors do not represent an accurate reproduction of the scene, but rather a combination of scene content and glare. We discuss the reasons why this unwanted addition of spread light cannot be removed from an acquired image. To this aim, we cast the problem of glare-removal into an estimation task and focus on the aspects that make the unfolding of glare an ill-posed or ill-conditioned problem—such as nonlinearity, information loss, or eye model uncertainty. For each mechanism of glare formation, we point to the corresponding influence in terms of ill-posedness and ill-conditioning of the problem. We do not aim at proposing or reviewing solutions to the glare problem but rather at identifying more precisely the challenges it poses.