LinkNet: Relational Embedding for Scene Graph
Objects and their relationships are critical contents for image understanding. A scene graph provides a structured description that captures these properties of an image. However, reasoning about the relationships between objects is very challenging and only a few recent works have attempted to solve the problem of generating a scene graph from an image. In this paper, we present a method that improves scene graph generation by explicitly modeling inter-dependency among the entire object instances. We design a simple and effective relational embedding module that enables our model to jointly represent connections among all related objects, rather than focus on an object in isolation. Our method significantly benefits main part of the scene graph generation task: relationship classification. Using it on top of a basic Faster R-CNN, our model achieves state-of-the-art results on the Visual Genome benchmark. We further push the performance by introducing global context encoding module and geometrical layout encoding module. We validate our final model, LinkNet, through extensive ablation studies, demonstrating its efficacy in scene graph generation.
Summary -------------  This paper proposes an attentional graph neural network approach for scene graph classification (classifying the objects in a scene, as well as the relationships between those objects) and applies it to the Visual Genome dataset, achieving state-of-the art performance with significant improvements particularly on the scene graph classification tasks.  Overall, I think this approach seems very promising and the results are quite impressive (though I should say I am less familiar with the literature on scene graph classification specifically). The architecture used in the paper is not particularly novel as graph neural networks have (1) previously been tried and (2) the paper mostly just borrows its architecture from the Transformer architecture (see below), but I think the impressive results can make up for a lack of novelty. However, I did find the paper somewhat difficult to read and it took me several passes through Section 3 to understand exactly what was going on. Additionally, I found the experiments to be a bit weak and some of the architectural choices to be somewhat arbitrary and not justified.  Quality ---------  Some of the choices of the architecture seem somewhat arbitrary and it's not clear to me their significance. For example, why is it necessary to turn O_4 into one-hot vectors via an argmax? Or, why is O_3 used in constructing E_0? Does these choices really change performance?  I appreciate the ablation experiments which show that the Global Context Embedding, Geometric Layout Encoding, and Relational Embedding are all important aspects, but I feel these experiments could have gone further. For example, the relational embedding is used four times: twice for node classification and twice for edge classification. Is it needed for both node and edge classification, or is it sufficient to just use it e.g. for node classification? And how much of an impact is there in using two stacked relational embeddings rather than one (or three)? Moreover, there are no ablation experiments which completely remove the relational embedding (but e.g. leave in the global context or the geometric layout encoding), making it difficult to judge how much that is improving things over a baseline.  Clarity --------  I found Section 3 hard to follow, with inconsistent and sometimes confusing notation. The following points were the hardest for me to follow:  - Sometimes the text specifies a variable name for a parameter matrix (e.g. W_1, U_1), and sometimes it uses an fc layer (e.g. fc_0, fc_1). I take it the difference is that the parameter matrices don't include bias terms, but I think it would be far easier to understand if the paper just used fc functions everywhere and specified somewhere else (e.g. in a footnote, or in the appendix) which fc terms used a bias and which did not. - I would strongly recommend a revision include (for example) an algorithm box which makes it clearer exactly how the computation is structured (this could even be in the appendix). - I found Figure 1 to be confusing, especially since there are terms used in the figure which aren't defined anywhere but the text (ideally, the figure should be interpretable in isolation). For example, O_3 and O_4 are not defined in the figure. I think the figure could be redone to be more explicit about where the different terms come from (I think there is a disproportionate amount of whitespace in the relational embedding and FC boxes, so these could be shrunk to make room for more detail). - Another detail which wasn't clear to me is where the features f_{i,j} (which are used in Eq. 12) come from. Are these the same ROI features introduced in 3.3.1? If so, how are they indexed by j? I thought f_{i} was of dimensionality 4096.  Originality -------------  There are two forms of novelty in this paper: first, applying the idea of attentional graph neural networks to scene graph classification, and second, including auxiliary losses and computations in the global context embedding and the geometric layout encoding. However, the architecture of the relational embedding that is used (Eqs. 1-3 and 4-5) is almost exactly that from the Transformer architecture (Vaswani et al., 2017). Moreover, the idea of using (non-attentional) graph neural networks in the context of scene graph classification was previously tried by Xu et al. (2017). Thus, while the results are impressive, there does not seem to be much innovation in the architecture.  Some other recent work has also used graph neural networks for just region classification and may be worth discussing and comparing to, see: Chen, X., Li, L., Fei-Fei, L., and Gupta, A. (2018). Iterative visual reasoning beyond convolutions. CVPR 2018.  Significance ----------------  As far as graph neural networks go, it is very interesting to see them applied more widely to more naturalistic data like raw images and I think this paper is significant in that regard (even if it is not the absolute first to do so). More significant are the results that the method achieves on the Visual Genome dataset, which seem to significantly push the state of the art.  Edit after author response: I have read the other reviews and the author response and am quite satisfied with the addition of several new ablation experiments. I do wish the authors had attempted to explain why they made the choices that they did about E_0 and O_4 in the first place (and hope that they will in the final revision), but it is good to see the empirical experiments showing those choices do matter. I also wish that the authors had explained a bit further how they see their work as different from previous approaches in the rebuttal, as suggested by R3, and hope that they will do so in the final version. All that said, I think the results are strong enough and the experiments now thorough enough that I have increased my score to a 7.