Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet [6]  are visualized at keypointnet.github.io.
The paper presents a self-supervised approach for detecting 3D key-points using end-to-end geometric reasoning.  Given two views of the same object with a known rigid transformation, the network is indirectly trained for keypoint localization by seeking key-points that best relate both views. An objective function is proposed that incorporates multiple domain-specific geometric cues to train the network. Multi-view consistency between the two sets of points under ground-truth transformation is used to enforce that the estimated key-points from both views can be transformed to each other via the known rigid transformation. To avoid degenerated solutions where all key-points collapse to the same location, two additional loss functions are introduced. The first loss measures the misfit between the ground-truth relative rotation and the estimated relative rotation using estimated key-points, while the second loss discourages proximity of the key-points in 3D. Finally, a silhouette consistency loss is introduced to ensure that the 2D projections of 3D key-joints lie inside the object silhouettes. Experiments are performed on the ShapeNet dataset, where the authors have obtained the annotations for three object categories via Mechanical Turk. Experimental evaluation demonstrates that the proposed self-supervised approach outperform supervised counterpart.   Pros: + The paper is well-written and easy to read. + The idea of discovering 3D key-point is very nice, though not completely novel (see below) + I like the idea of relative pose estimation as a supervisory signal.  + The qualitative results look quite good.  + The proposed approach can be extended to real applications such human pose estimation and hand pose estimation. The used supervisory signals for human pose estimation are also available, for example, in HUMAN3.6M dataset or for hand pose estimation in CMU's Panoptic Studio.  Cons: - The idea of self-supervised 3D keypoint localization is not entirely novel and have been addressed in the literature. For example in [a].  - I would have liked to see quantitative ablative studies in the paper, that how much each loss contributes to the final performance. At the moment this is only provided in the supp. material without any quantification.  - I would have liked to see some results on realistic data such as for HUMAN3.6M or some other application where same supervisory signals can be used.    Summary: I think it is a good paper with sufficient novel contributions to be accepted to NIPS. The proposed method can be extended to many other applications.   Some references that should be added:  [a] Tung et al., Self-supervised Learning of Motion Capture, NIPS'17 [b] Rohdin et al., Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation, CVPR'18 [c] Molchanov et al., Improving Landmark Localization with Semi-Supervised Learning, CVPR'18 [d] Zhang et al., Unsupervised Discovery of Object Landmarks as Structural Representations, CVPR'18  or other recent CVPR'18 papers that I might have missed.    The rebuttal addressed most of my concerns. It still misses experiments on a real application, but I agree with the authors that it is an interesting future direction.