Learning Conditioned Graph Structures for Interpretable Visual Question Answering
Visual Question answering is a challenging problem requiring a combination of concepts from Computer Vision and Natural Language Processing. Most existing approaches use a two streams strategy, computing image and question features that are consequently merged using a variety of techniques. Nonetheless, very few rely on higher level image representations, which can capture semantic and spatial relationships. In this paper, we propose a novel graph-based approach for Visual Question Answering. Our method combines a graph learner module, which learns a question specific graph representation of the input image, with the recent concept of graph convolutions, aiming to learn image representations that capture question specific interactions. We test our approach on the VQA v2 dataset using a simple baseline architecture enhanced by the proposed graph learner module. We obtain promising results with 66.18% accuracy and demonstrate the interpretability of the proposed method. Code can be found at github.com/aimbrain/vqa-project.
-------------- Summary: --------------  The submission proposed a graph-based, question-image fusion mechanism for visual question answering. This approach learns an embedding of question and object detection features and uses it to predict question-image dependent adjacency matrices -- essentially a graph structure with nodes corresponding to object detections and edges weighted according to this learned metric. This learned module makes up the core technical contribution of the work as both spatial GCNs and the use of object detection features come from existing work.    -------------- Clarity: --------------  [C1] The portions of the method discussed in the submission are relatively clearly described. However, the model description stop abruptly after computing the GCN node representations. relying instead on Figure 2 and L92/L201 to convey the rest of the model (max-pool followed by full-connected layers). While these remaining components are quite simple, the choice of performing a max-pool is interesting but remains unjustified in the current approach.  [C2] The baselines are described incredibly quickly and without sufficient detail for the 'attentional' model to understand the comparison made in Table 1.  [C3] Significant space in the submission is used to display multiple large example images, Fig 1, 3, and 4, but there is limited discussion of these figures. Further, I'm not sure what are the take-aways from the examples in Fig 3 and 4. The authors point to these images as examples of interpretability, but the graph structure (the novel addition over standard attentional frameworks) doesn't seem to clearly be very interpretable.   [C4] Many of the figure captions are uninformative or incomplete.  [C5] The paper does not achieve state of the art results as claimed. I don't particularly find small improvements in SoTA to be that good of a measure of a paper anyway, but given that the claim appears in multiple places in the submission, it is worth pointing out.   -------------- Quality: --------------  [Q1] I'm all for the diversity of ideas and feel the fact that this submission performs similarly (or somewhat worse than) existing work while introducing a significantly different attentional mechanism is valuable. That said, given the lack of analysis or discussion it is difficult to find insights about the problem from this work.  [Q2] The proposed graph learning approach is not well explored. It is unclear what sort of spatial relationships the Guassian GCN kernels are learning. It is unclear what question-conditioned object relationships the graph learner module is representing. It is not discussed what effect the top-k or max-pool design decisions have on the model. It would be good if authors could provide some discussion of these.  [Q3] I do not see sufficient discussion or experiments of interpretability to substantiate the claims that the graph based attention model is more interpretable. It would be good for the authors to define what they mean by interpretable and provide some analysis based on human studies.  -------------- Originality: --------------  [O1] Some related work on attentional GCNs (i.e GCNs which learn edge weights) is missed. For example, GRAPH ATTENTION NETWORKS from ICLR 2018. I admit that ICLR and this submission deadline are fairy close (~2 weeks) but this work has been publicly available for some time.  [O2] The core novelty of this work is in the prediction of question-conditioned edge weights between object nodes in the graph structure. To my knowledge, this is fairly novel in the context of VQA.  -------------- Significance: --------------  [S1] Without additional analysis to identify the behavior, strengths, and weaknesses of the proposed method, I don't think the current submission would significantly advance understanding on the topic of VQA or question-conditioned attention mechanisms. Which is unfortunate given the direction itself may be valuable.  Post Rebuttal: The author response addressed some of my concerns and I've updated my recommendation accordingly. I urge the authors to put concerted effort into explore and exposing all the interesting design decisions and model behaviors of their new architecture in the final draft. The proposed solution is a fairly significant shift from existing approach that performs fairly similarly and it is useful to the community to illuminate why.