High-Order Attention Models for Visual Question Answering
The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they take into account different data modalities, such as visual and textual input. In this paper we propose a novel and generally applicable form of attention mechanism that learns high-order correlations between various data modalities. We show that high-order correlations effectively direct the appropriate attention to the relevant elements in the different data modalities that are required to solve the joint task. We demonstrate the effectiveness of our high-order attention mechanism on the task of visual question answering (VQA), where we achieve state-of-the-art performance on the standard VQA dataset.
The paper presents a novel method to model attention on the image and the question for the task of Visual Question Answering (VQA). The main novelty lies in proposing a generic formulation of attention for multiple modalities – a combination of unary, pairwise and ternary potentials for 3 modalities (image, question and answer) in VQA. The proposed approach is evaluated on the VQA dataset and outperforms existing models when ternary potentials are used. 
Strengths:
1.	A generic formulation of attention for multimodal problems that is not specific to a particular task is useful because especially if it can be shown to generalize to more than one task. 
2.	The ablation study showing the performance without ternary potential is useful to understand how much ternary potential helps.
3.	The paper is clearly written.
Weaknesses:
1.	The approach mentions attention over 3 modalities – image, question and answer. However, it is not clear what attention over answers mean because most of the answers are single words and even if they are multiword, they are treated as single word. The paper does not present any visualizations for attention over answers. So, I would like the authors to clarify this.
2.	From the results in table 1, it seems that the main improvement in the proposed model is coming from the ternary potential. Without the ternary potential, the proposed model is not outperforming the existing models for 2 modalities setup (except HieCoAtt). So, I would like the authors to throw light into this.
3.	Since ternary potential seems to be the main factor in the performance improvement of the proposed model, I would like the authors to compare the proposed model with existing models where answers are also used as inputs such as Revisiting Visual Question Answering Baselines (Jabri et al., ECCV16).
4.	The paper lacks any discussion on failure cases of the proposed model. It would be insightful to look into the failure modes so that future research can be guided accordingly.
5.	Other errors/typos:
a.	L38: mechanism -> mechanisms
b.	L237 mentions that the evaluation is on validation set. However Table 1 reports numbers on the test-dev and test-std sets?
Post-rebuttal comments:
Although the authors' response to the concern of "Proposed model not outperforming existing models for 2 modalities" does not sound satisfactory to me due to lack of quantitative evidence, I would like to recommend acceptance because of the generic attention framework for multiple modalities being proposed in the paper and quantitative results of 3-modality attention outperforming SOTA. The quantitative evaluation of the proposed model's attention maps against human attention maps (reported in the rebuttal) also looks good and suggests that the attention maps are more correlation with human maps' than existing models. Although, we don't know what this correlation value is for SOTA models such as MCB, I think it is still significantly better than that for HieCoAtt.
I have a question about one of the responses from the authors --
> Authors' response -- “MCB vs. MCT”: MCT is a generic extension of MCB for n-modalities. Specifically for VQA the 3-MCT setup yields 68.4% on test-dev where 2-layer MCB yields 69.4%. We tested other combinations of more than 2-modalities MCB and found them to yield inferior results.
Are the numbers swapped here? 69.4% should be for 3-MCT, right? Also, the MCB overall performance in table 1 is 68.6%. So, not sure which number the authors are referring to when they report 68.4%.