MarrNet: 3D Shape Reconstruction via 2.5D Sketches
3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.
This paper describes a technique for estimating voxelized 3D shapes from single images. As directly predicting 3D shapes from images is hard, the paper proposes to separate the problem into 2 tasks - inspired by Marr's theory about vision. In the first part the method takes the image as input and predicts several intrinsic quantities ("2.5D sketch"), in particular depth, surface normals and the 2D silhouette using an encoder-decoder architecture. This information is fed into a second encoder-decoder network which predicts the volumetric 3D representation at 128^3 resolution. The advantage of the method is that the second part can be trained on synthetic data allowing for "domain transfer" based on the 2.5D sketches which are invariant to the actual appearance. Experiments (mainly qualitative) on Shapenet, IKEA and Pascal3D+ are presented.
Overall, I liked the idea. The paper is clear, well motivated and written and the qualitative results seem convincing. However, I have some concerns about the fairness of the evaluation and the self-supervised part which I like to have answered in the rebuttal before I turn towards a more positive score for this paper.
Positive aspects:
+ Very well written and motivated
+ Relevant topic in computer vision
+ Interesting idea of using intermediate intrinsic representations to facilitate the task
+ Domain transfer idea is convincing to me (though the self-supervision and fine-tuning of the real->sketch network are not fully clear to me, see below)
+ Qualitative results clearly superior than very recent baselines
Negative aspects:
- Self-supervision: It is not clear to me why this self-supervision should work. The paper says that the method fine-tunes on single images, but if the parameters are optimized for a single image, couldn't the network diverge to predict a different shape encapsulated in the shape space for an image depicting another object while still reducing the loss? Also, it is not very well specified what is finetuned exactly. It reads as if it was only the encoder of the second part. But how can the pre-trained first part then adapted to a new domain? If the first part fails, the second will as well.
- Domain adaptation: I agree that it is easy to replace the first part, but I don't see how this can be trained in the absence of real->sketch training pairs. To me it feels as the method profits from the fact that for cars and chairs pretraining on synthetic data already yields good models that work well with real data. I would expect that for stronger domain differences the method would not be able to work without sketch supervision. Isn't the major problem there to predict good sketches from little examples?
- Baselines: It is not clear if the direct (1-step) baseline in the experiments uses the same architecture or at least same #parameters as the proposed technique to have a fair evaluation. I propose an experiment where the same architecture is used but insteads of forcing the model (ie, regularizing it) to capture 2D sketches as intermediate representations, let it learn all parameters from a random initialization end-to-end. This would be a fair baseline and it would be interesting to see which representation emerges. Another fair experiment would be one which also has 2 encoder-decoder networks and the same number of parameters but maybe a different distribution of feature maps across layers. Finanlly a single encoder-decoder architecture with a similar architecture as the proposed one but increased by the number of parameters freed by removing the first part would be valuable. From the description in the paper it is totally unclear what the baseline in Fig. 4. is. I suggest such baselines for all experiments.
- DRC: It should be clarified how DRC is used here and if the architecture and resolution is adopted for fairness. Also, DRC is presented to allow for multi-view supervision which is not done here, so this should be commented upon. Further the results of the 3D GAN seem much more noisy compared to the original paper. Is there maybe a problem with the training?
- Quantitative Experiments: I do not fully agree with the authors that quantitative experiments are not useful. As the paper also evaluates on ShapeNet, quantitative experiments would be easily possible, and masks around the surfaces could be used to emphasize thin structures in the metric etc. I suggest to add this at least for ShapeNet.
Minor comments:
- Sec. 3.1: the deconvolution/upconvolution operations are not mentioned
- It is unclear how 128^3 voxel resolution can be achieved which typically doesn't fit into memory for reasonable batch sizes. Also, what is the batch size and other hyperparameters for training?
- Fig. 8,9,10: I suggest to show the predicted sketches also for the real examples for the above mentioned reasons.