Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
Understanding the 3D world is a fundamental problem in computer vision. However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved. In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective. We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the perspective transformation. More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision. We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes. Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.
The paper proposes an alternative/additional loss for the task of single image object reconstruction - that the projections of the predicted and ground-truth 3D shape should agree across various views. This is cleanly implemented in a CNN framework using spatial transformers and the paper empirically demonstrates the benefits of using this loss instead of / in addition to a conventional loss.       
         I like the motivation of the paper that in case full 3D information is not available, we can still train a volume prediction CNN using an alternative loss which enforces silhouette consistency with available observations (eg. can be used on Ebay/Amazon etc.). The implementation proposed in the paper (using spatial transformers and max-projection on predicted volume to obtain a silhouette) is also clean, intuitive and can be easily adoptable.          My main concerns are regarding the experimental setup and results (explained in detail below)          Experimental Results : A central point in the experiments is that using the projection loss gives better reconstructions as well as more generalizable models - I don't understand why that should be true. Basically, the projection loss is just an alternative loss which instead of directly penalizing distance between predicted and true models, instead penalizes difference in their projections. It is not clear why this a better loss or why it adds any value at all in model-comb (except perhaps as an additional regularizer). However, the experiments show a clear gain over model-vol (chair in Table 1, generalization in Table 2) - one possible explanation for this is the evaluation metric (IU @ 0.5). Since the models are trained independently, the threshold 0.5 is really arbitrary (as different models may be calibrated differently) and I suspect having max-projection and loss with silhouette makes it a better threshold - for eg., the predictions by VO in Fig 3 are not well formed, but they would be if a lower threshold were used. I strongly recommend that the following experiment be run for the rebuttal - instead of measuring IU @ 0.5, the evaluation metric should be IU @ t where the threshold t is selected independently for each model (and category). If the conclusions presented still hold, I would be happy to update my assessment but otherwise I think this is a serious concern as I find it extremely counter-intuitive that the projective loss would help in training better/more general reconstruction models.          Experimental Setup : Despite the concerns above, I agree that the projection-loss based strategy is useful as it allows one to train without having the underlying 3D model available. Unfortunately, the experimental setup in the paper does not present this strongly - as L172 states, all the 24 views of the object are used for training the CNN and while technically the full 3D model is not available, it practically is. Infact recent experiments in the community (eg. Multi-view CNN) show that representing a 3D model by a collection of views instead of voxels performs even better for category classification. The point here being that using a diverse set of views essentially has similar information as using the full 3D model. If the experiments were to use a more natural setting where for each 3D model, only a few views from a narrow range of azimuth were available (which is arguably a better model for a learning agent which moves around objects) and still obtain comparable reconstruction, it would be a much stronger demonstration of the utility of the technique presented here.          Overall, while the central idea is interesting and novel with a clean implementation, there are concerns regarding both the conclusions mentioned - 1) the experimental setup used here is not really representative of a secnario with 'ground-truth 3D volume not available' and 2) It is unclear if the generalization is better die to the the alternative loss or is an artifact of the metric - hopefully the experiment suggested above would clarify (though the question of why it is better would still remain).          Minor Concerns/Suggestions :         - Technically, the encoder+decoder is supposed to learn a 3D view invariant output and that should just emerge as training progresses, and it should not be necessary to enforce anything on the encoder. The pre-training described in L134-144 seems adhoc. Also, it's unclear why using the deep-rotator encoder does leads to view-invariance since it's not explicitly enforced during the deep-rotator training.          - Using L1 loss instead of L2 in Eqn 2 typically leads to better reconstructions.          - While the overall writing of the paper was great, the introduction using the conscious learning agent seems contrived. As mentioned above, an active agent would not explore all diverse views of the car, would not have a common canonical frame across all cars (and so not know the absolute pose of the car - only relative pose of different views) etc.  ----- Update after Rebuttal : The authors addressed the major concerns above by showing the conclusions hold with the suggested evaluation. An additional experiment with a restricted set of training views available for each object was also presented and I feel this better justifies the motivation of the paper.