We propose an approach to predict the 3D shape and pose for the objects present in a scene. Existing learning based methods that pursue this goal make independent predictions per object, and do not leverage the relationships amongst them. We argue that reasoning about these relationships is crucial, and present an approach to incorporate these in a 3D prediction framework. In addition to independent per-object predictions, we predict pairwise relations in the form of relative 3D pose, and demonstrate that these can be easily incorporated to improve object level estimates. We report performance across different datasets (SUNCG, NYUv2), and show that our approach significantly improves over independent prediction approaches while also outperforming alternate implicit reasoning methods.
<Summary>: This paper presented a method for incorporating binary relationship between objects (relative location, rotation and scale) into single object 3d prediction. It is built on top of previously published work of [a] and used same network architecture and loss as of [a] and only added the binary relations between objects for object 3d estimation. The results are shown on SUNCG synthetic dataset and only *4 image* instances of NYUv2 dataset which is very small for a computer vision task.
[a] Shubham Tulsiani, Saurabh Gupta, David Fouhey, Alexei A Efros, and Jitendra Malik. Factoring shape, pose, and layout from the 2d image of a 3d scene. In CVPR, 2018.
<Pros>: The paper tackles a problem of obvious interest to computer vision research community. It shows better results compared to previous similar work of [a] without considering binary relation between objects.
<Cons>:
*Technical details are missing:
The set of known and unknown variables are not clear throughout the paper:
-The extrinsic camera parameters are known or estimated by the method?
-The intrinsic camera parameters are known or estimated by the method?
-What are the properties of ground truth bounding boxes in 2D camera frame and 3D space?
-What is the coordinate of translation? is it in camera coordinate or world coordinate?
-What are the variations of camera poses in training and testing for synthetic dataset and how are the samples generated? Are the train/test images generated or are rendered images from previously published work of [b] used?
[b] Yinda Zhang, Shuran Song, Ersin Yumer, Manolis Savva, Joon-Young Lee, Hailin Jin, and Thomas Funkhouser. Physically-based rendering for indoor scene understanding using convolutional neural networks. In CVPR, 2017.
*The proposed method is trained on synthetic dataset of SUNCG and their object relations have biases from scene creators. While using binary relation between objects increase the recall in prediction it can also make the predictions bias to the most dominant relations and decrease the precision of detection in rare cases in synthetic dataset. Also, such bias can decrease prediction precision in images of real scenes.
 
*One of the main issues in this paper is that the result of fully automated pipeline versus having ground-truth annotation at test time are mixed up. For example, in the teaser figure (Figure 1-b), does the proposed method use ground truth bounding boxes or not? It is mentioned in figure caption: “(b) Output: An example result of our method that takes as input the 2D image and generates the 3D layout.”. Is the input only 2D image or 2D image + ground truth object bounding boxes?
In order to make sure that reader understands each qualitative result, there should be a column showing the “Input” to the pipeline (Not “Image”). For example, in Figure 3 and Figure 4, the image overlaid with input ground-truth bounding boxes should be shown as input to the algorithm. 
*The experiments and results does not convey the effectiveness of the proposed approach. There are major issues with the quality of the experiments and results. Here are several examples:
- Missing baseline: Comparison with the CRF-based baseline is missing. This statement is not convincing in the introduction: “One classical approach is to use graphical models such as CRFs. However, these classical approaches have usually provided little improvements over object-based approaches.” For a fair comparison with prior works, reporting results on a CRF-based baseline using similar unary predictions is necessary. 
-The experimental results are heavily based on ground truth boxes for the objects, but it is not clear how/where the ground truth boxes are given at the test time and which part is actually predicted.
-If the ground truth boxes are given at the test time, it means that the ground truth binary relations between objects are given and it makes the problem trivial.
-It is not clear what is the ground truth box in experimental setup. Is it amodal object box or the ground truth box contains only the visible part of the object? 
-The qualitative results shown in Figure 4 have full objects in voxel space with predicted rotation, scale and translation. In the qualitative result of Figure 3 and Figure 5 the voxel prediction is shown as final output. Why the result of full object in voxel space with predicted (rotation, scale and translation) is not shown in Figure 3 and Figure 5 and why it is shown in Figure 4?
*Very limited results on real images:
-Quantitative result on a dataset of real images is missing. The results on synthetic datasets is not a good proxy for the actual performance of the algorithm in real use cases and applications.
- The paper only shows few results of NYUv2 on known ground truth boxes. The errors in object detection can be propagated to the 3D estimation therefore these qualitative results are not representative of the actual qualitative performance of the proposed algorithm. Several randomly selected qualitative results on a dataset of real images “without ground-truth boxes” are needed for evaluating the performance of the proposed method on real images. 
-Reporting variation in all parameters of scale, rotation and translation is necessary in order to find the difficulty of the problem. For example, what is the distribution of object scale in different object categories. What is the error of scale prediction of we use mean object scale for each object category for all object instance at test set?
*Unclear statements and presentation:
- It is mentioned in the paper: “While the incorporation of unary and relative predictions can be expressed via linear constraints in the case of translation and scale, a similar closed form update does not apply for rotation because of the framing as a classification task and non-linearity of the manifold.”
-Is it necessary for the relative rotation to be formulated to classification task? 
-If not the comparison of modeling relative rotation via linear constraints is missing.
- In some of the tables and figures the “know ground-truth boxes/detection setting” are in bold face and in some cases are not. This should be consistent throughout the paper.