Mapping Images to Scene Graphs with Permutation-Invariant Structured Prediction
Machine understanding of complex images is a key goal of artificial intelligence. One challenge underlying this task is that visual scenes contain multiple interrelated objects, and that global context plays an important role in interpreting the scene. A natural modeling framework for capturing such effects is structured prediction, which optimizes over complex labels, while modeling within-label interactions. However, it is unclear what principles should guide the design of a structured prediction model that utilizes the power of deep learning components. Here we propose a design principle for such architectures that follows from a natural requirement of permutation invariance. We prove a necessary and sufficient characterization for architectures that follow this invariance, and discuss its implication on model design. Finally, we show that the resulting model achieves new state-of-the-art results on the Visual Genome scene-graph labeling benchmark, outperforming all recent approaches.
This paper studies the property of permutation invariance in the context of structured prediction. The paper argues that in many applications permutation invariance is a desirable property of a solution and it makes sense to design the model such that it is satisfied by construction rather than to rely on learning to get this property. The paper proposes a model to represent permutation invariant functions and claims that this model is a universal approximator within this family. The proposed method is evaluated on a synthetic and a real task (labelling of scene graphs).  1) Most importantly, I think that in the current form the proof of the main theoretical result (Theorem 1) is wrong. The problem is with the reverse direction proving that any permutation invariant function can be represented in the form of Theorem 1. Specifically, Lines 142-159 construct matrix M which aggregates information about the graph edges. However, the actual function \rho is defined (line 160) without taking the matrix M into account. Instead, it looks at y, which is the output of the function. As is, the statement in fact says "define the function output with y, now set it equals to y", which is a logical error and does not prove anything.  2) I also think that the assumptions that z_k uniquely identifies the node (line 138) and that F is a function of only pairwise features (line 140) contradict each other, because if there are no unary features they cannot identify the nodes.  3) The form of the function F defined in Theorem 1 is very similar to one round of neural message passing [9] and thus will not allow to propagate information across the graph. For example, imaging the toy situation: one node has a very strong preference for one label through the unary potentials, the others have no preferences. The pairwise potentials defined a long chain of equalities saying that in the end of the day all the variables should take the same value. However, to infer this we need to propagate information across the whole graph, which will need many rounds.  4) The experimental evaluation is somewhat interesting, but does not have sufficient description to be reproduced and does not show significant improvements, thus I find it not sufficient for acceptance.  Minor: Line 147 probably should have the sum over j and not over pairs i, j   ==== after the response 1) I agree with the authors my point 1 was not valid. 2) I agree with the authors that this particular place has no contradiction. However, I'm still not quite buying the whole argument based on the perfect hash function. The theorem is proving a statement about continuous inputs, which means that for any hash function with a finite hash table there exist a pair of inputs z_i, z_j, for which there will be a collision. The construction will break for this input. I think we cannot use a different hash function for each input, but need it to defined before looking at the inputs, which looks like a fundamental flaw in the proof technique. 3) I'm not getting why the presented method cannot be viewed as a message passing algorithm. Functions phi can be viewed as messages and alpha as the message aggregation. The big difference is probably the fact that the messages are not passed w.r.t. the initial graph, but w.r.t. the complete graph on the nodes. 4) I still think that the description of the experiments does not allow reproducibility and the improvements are somewhat incremental (the response claims otherwise, but does not give any extra data, e.g. some upper bounds).  Having said that, I'm ready to raise my score to 5, because the theorem might be correct, but the paper still requires a major revision (proof of the theorem and the description of the experimental setup).