Unsupervised Learning of Object Landmarks through Conditional Image Generation
We propose a method for learning landmark detectors for visual objects (such as the eyes and the nose in a face) without any manual supervision. We cast this as the problem of generating images that combine the appearance of the object as seen in a first example image with the geometry of the object as seen in a second example image, where the two examples differ by a viewpoint change and/or an object deformation. In order to factorize appearance and geometry, we introduce a tight bottleneck in the geometry-extraction process that selects and distils geometryrelated features. Compared to standard image generation problems, which often use generative adversarial networks, our generation task is conditioned on both appearance and geometry and thus is significantly less ambiguous, to the point that adopting a simple perceptual loss formulation is sufficient. We demonstrate that our approach can learn object landmarks from synthetic image deformations or videos, all without manual supervision, while outperforming state-of-the-art unsupervised landmark detectors. We further show that our method is applicable to a large variety of datasets -faces, people, 3D objects, and digits -without any modifications.
Summary: This paper proposes a method for conditional image generation by jointly learning "structure" points such as face and body landmarks. The authors propose to use a convolutional neural network with a modified loss to capture the image transformation and landmarks. They evaluate their approach on a set of datasets including CelebA, VoxCeleb, and Human 3.6M.  Positive:  -The problem addressed is an important problem and the authors attempt to solve it using a well engineered approach. -The results seem to be competitive to some state of the art approaches.  Negatives: -The pre-processing using heat maps, normalizing them into probabilities, then using a gaussian kernel to produce the features is a bit heuristic.  -The claim in P4 L132-135 saying this is usually done using GANs, it would have been good to show some results and comparison against GAN based methods. Comparison against [37] for image generation using landmark/control/structure points would have helped. -The use of two networks, landmark network and image generation network, rather than training a joint network seems inefficient.  Landmarks and image generation are two complimentary that could be modeled in a joint network.