We study the effect of using different object information for visual relation detection with a focus on depth maps. State of the art visual relation detection methods mostly rely on object information extracted from RGB images such as predicted class probabilities, 2D bounding boxes and feature maps. In this paper, we argue that the 3D positions of objects in space can provide additional valuable information about object relations. This information helps not only to detect spatial relations, such as standing behind, but also non-spatial relations, such as holding. Since 3D information of a scene is not easily accessible, we incorporate a pre-trained RGB-to-Depth model within a visual relation detection framework and release a new dataset VG-Depth as an extension to Visual Genome. We note that given the highly imbalanced distribution of relations in the available datasets, typical evaluation metrics for visual relation detection cannot reveal improvements of under-represented relations. To address this problem, we propose using Macro Recall@K for visual relation detection. Our experiments confirm that the performance of state-of-the-art visual relation detection approaches can significantly be improved by utilizing depth map information.
Relational learning is an established research area in machine learning. State-of-the-art approaches (Nickel et al., 2016) describe relations as triples of the form (subject, predicate, object), such as (Man, rides, Bike). The triples, which all together form a knowledge graph, are typically extracted from structured data such as the infoboxes in Wikipedia and other sources.
In the last years, the detection of relations from images, i.e., visual relation detection, has also gained a lot of interest. One reason is that understanding relations between entities can play an important role in decision making. For example, detecting whether a man is on a bike or next to a bike is a crucial challenge in autonomous driving. State-of-the-art works in this area utilize information of objects in the scene such as class labels, bounding boxes and RGB features, to capture pairwise relations using different models. In this paper, we argue that relation detection can additionally benefit from objects' 3D information. This information can help to distinguish between many relations such as behind, in front of and even improve detection in situations where the objects are nearby such as covered in.
Unfortunately, most available datasets, specifically the ones with relational annotations such as Visual Relation Detection (VRD) (Lu et al., 2016) and Visual Genome (VG) (Krishna et al., 2017), lack this 3D information since acquiring them is a cumbersome task requiring specialized hardware. We solve this issue by synthetically generating the corresponding depth maps of images in these datasets. Depth maps provide the objects' distance from the camera and the availability of large corpora of RGB-D pairs, e.g. NYU-Depth-v2 (Nathan Silberman & Fergus, 2012) dataset enables us to learn the mapping between any RGB image to its corresponding depth map using a fully convolutional neural network. We can then apply the trained network on images from VG, converting them into depth maps. We call this dataset extension VG-Depth 1 .
The extracted features from depth maps, together with other object information extracted from the RGB images, are the basis for relation detection in our framework. We note that the typically employed Recall@K metric, which we call as Micro Recall@K, cannot properly reveal the improvements of under-represented relations in highly imbalanced datasets such as Visual Genome. This might be an issue in applications such as autonomous driving where it is important to ensure that the model is capable of predicting also more important but less represented predicates such as walking on Figure 1: An image from the Visual Genome dataset showing several men sitting next to each other and the corresponding generated depth map. Bright colors indicate a larger distance to the camera. Knowing the depth of, e.g., the individuals, the house and the car, plays an important role in detecting relations such as next to and behind.
(648 in VG test set) and not just wearing (20,148 in VG test set). We tackle this issue by proposing to employ Macro Recall@K, where the Recall@K is computed per predicate class and averaged over, thereby eliminating the effect that over-represented classes have in Micro Recall@K setting. Reporting on both Macro@K and Micro@K, we study the effect of each object information on visual relation detection.
In summary, our contributions are as follows:
1. We perform an extensive study on the effect of different object information in visual relation detection, including 3D information. We show in our empirical evaluations using the VG dataset, that our model can outperform competing methods by a margin of up to 8% points, even those with contextualization or using information extracted from external language sources. 2. We propose using Macro Recall@K to better reveal the improvements in visual relation detection when dealing with highly imbalanced datasets such as Visual Genome. 3. To compensate for the lack of 3D data in Visual Genome, we use an RGB-to-Depth model trained on separate corpora of available pairs in both modalities and apply it to VG images and releasing a new dataset, VG-Depth.
In general, in this work we aim to answer the following questions:
1. If we are given only depth maps of unknown objects in a scene, how accurately can we infer the distribution of possible pairwise relations? How do other sources of object information compare to it? 2. Current visual relation detection frameworks commonly rely on extensive object information such as class labels, bounding boxes, RGB features, contextual information, etc. Does it bring any additional information, if we also include depth representations in these frameworks or would that only bring redundant knowledge about the scene? 3. Do the current evaluation metrics properly reflect the improvements of under-represented relations within highly imbalanced datasets?
********* Post Rebuttal *********
I appreciate the authors' effort in providing thorough responses and revised manuscript. 
I agree with the authors that "the finding not being surprising" is not a ground for rejection. I tried to word my final decision carefully but it seems it has still caused confusion for the authors. As I have mentioned in my original review, the rating was a result of the 4 points considered *together* . 
That is, if one exploits privileged information that needs extra sensory data and/or annotation (point 2), *and*, this privileged information is clearly related and thus should be normally useful for the final task (point 1), *and* achieve marginal improvements (point 3), it can be a ground for rejection. Especially, given that prior works with similar arguments exists (point 4). 
The rebuttal has alleviated the issue of marginal improvements (point 3) by introducing meanR@K (or as the revised paper refer to it, Macro R@K). Here, the improvements are more significant both compared to the state of the art and ablated baselines.
The authors also argue that the related [Yang et al. 2018] paper (point 4) should be considered a concurrent submission since the authors original submission was to AAAI18. 
The rebuttal also addresses other clarity or experimental issues which improves the quality of the revised work.
Finally, I understand that the privileged information is only required during training time which is a good point.
All in all, *assuming that [Yang et al. 2018] is considered a concurrent work* according to ICLR, I think the revised paper becomes slightly above borderline and thus I change my rating to "weak accept". If [Yang et al. 2018] is not considered concurrent work, then, a conclusive comparison is required for the acceptance of the current work.
********* Summary *********
 
The paper poses the question of whether depth information is informative for visual relationship prediction using still images. It is intuitive that 3D arrangement of objects in an image can be a useful cue for predicting their relationship. As such it is important to see whether and to what extent depth information complements RGB information for visual relation detection. That is the focus of this paper.
The paper proposes to use an off-the-shelf monocular depth estimation networks to augment the available RGB information towards better visual relation detection. For that, it proposes a specific network two-stream structure working on RGB image and (predicted) depth image. The proposed model demonstrates improved results upon state of the art for visual relation prediction.
 
********* Strengths and Weaknesses *********
 
+ A comprehensive set of tests has been conducted. 
+ Zero-shot prediction results are particularly interesting.
+ The experiment on ranking the predicate classes based on the change in prediction accuracy before and after using depth information (Figure 4) is interesting and intuitive.
* The final results improve upon the state-of-the-art, especially on the zero-shot learning regime. However, it seems that the improvement is mainly coming from the new architecture as opposed to the inclusion of the depth information. That is, ours_{c,v,l} brings most of the improvement already the last step to ours_{c,v,l,d} is negligible for non-zero-shot case.
- Along the same line, it’s possible that this small difference between ours_{c,v,l} and ours_{c,v,l,d} for the standard predicate prediction, can be due to a hyper-parameter optimization that is (only or more thoroughly) done for ours_{c,v,l,d}. The hyper-parameter optimization scheme for different baselines is not described. 
- Given the small difference of ablation levels, the comparison will be stronger if done multiple times and reporting mean and standard deviation of the results.
- For a fair comparison the visual feature vector v_{so} should be tried as the feature of the union bound box of both subject and object same way as it is done for depth feature vector d_{so}. 
- The paper refers to “Ours-d’_{so}” as a baseline that *only* uses depth information with no image/label information. However, it seems that the region proposals for this feature are coming from the image-based network that uses image information. 
 
- Important related but uncited works:
(1) [“Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information” ECCV workshops 2018] studies the same question as part of their work.
 
********* Final Decision *********
 
I do not find the paper passing the acceptance bar mainly due to the following reasons together:
1) The finding is not surprising since most of the visual relations are either explicitly depth-related (e.g., behind) or are semantically constrained by depth (e.g. riding cannot happen at different depths when the image is taken orthogonal to the rider).
2) an additional depth dataset is used which provides the model with privileged information. Should it have been the case that depth information were inferred without an additional offline dataset, the results would have been more interesting.
3) the improvements due to the additional depth network are not significant or conclusive.
4) there is a prior uncited work with the same research question for effectiveness of depth information in visual relation detection which uses a similar approach.
 
********* Minor points *********
- the code is not available. This is especially important since the paper is outperforming prior works which could be a contribution if reproducible.
- Section 2.2: is l_{so} concatenation of l_s and l_o?
- Section 2.2: y_{spo} is defined but never used.
- Equation 2: why do we have both e_p and f in the exponents? Aren’t they the same?
- Equation 2:  P is never defined.
- Page 5: “a fully connected hidden layer of 64, 200, 4096 and 20 neurons”: this amounts to 3 hidden layers.
- Why VGG network for visual feature and AlexNet for depth features?
- zero-shot learning results on visual genome is missing
- training procedure is a bit unclear: the text suggest that the fine tuning and/or learning of the three components might happen separately. It is important to clearly state if they are done in an end-to-end fashion and simultaneously or separately; and why.
- It’s good to name the method in table 2 in the same fashion as table 1. With the current naming (based on architecture) it is a bit confusing to understand the content without additional cross referencing. For instance AlexNet-BN - Raw seems to correspond to Ours_{c,v,l,d}
- Figure 4: the frequency represented as different shades of red or blue is really hard to notice especially on a printed paper. The red vs blue color coding is not necessary since the bars going up or down indicate the same quality. So, it might be better to use red/blue for frequency instead (e.g. dark red high frequency to dark blue low frequency)
- Section 3.2: the AlexNet reference seems wrong, it should be "ImageNet Classification with Deep Convolutional Neural Networks" NIPS , 2012
- The structure of section 3.5 is currently flat while the content seems to be nested (two experiments and two sets of corresponding discussions). It will read better if they are organized into subsections.
 
 
********* Points of extensions (improvement) *********
- I believe *unsupervised* discovery of depth information for visual relation detection can be an interesting direction since it is not limited to the availability of relevant depth dataset. 
- It is not clearly motivated why one should use two separate networks for depth and RGB inputs in light of the additional complexity. For instance, it is good to discuss what is the advantage of the proposed (computationally more expensive) method over the following two simpler baselines:
- Faster RCNN is used on RGBD input to produce a single feature vector
- above case with RGB input but have the Faster RCNN predict the depth map as an auxiliary loss.