Bilinear Attention Networks
Attention networks in multimodal learning provide an efficient way to utilize given visual information selectively. However, the computational cost to learn attention distributions for every pair of multimodal input channels is prohibitively expensive. To solve this problem, co-attention builds two separate attention distributions for each modality neglecting the interaction between multimodal inputs. In this paper, we propose bilinear attention networks (BAN) that find bilinear attention distributions to utilize given vision-language information seamlessly. BAN considers bilinear interactions among two groups of input channels, while low-rank bilinear pooling extracts the joint representations for each pair of channels. Furthermore, we propose a variant of multimodal residual networks to exploit eight-attention maps of the BAN efficiently. We quantitatively and qualitatively evaluate our model on visual question answering (VQA 2.0) and Flickr30k Entities datasets, showing that BAN significantly outperforms previous methods and achieves new state-of-the-arts on both datasets.
The main contribution of this paper is a multimodal attention that links previous work of low-rank pooling, with Co-Attention methodologies, i.e., use of affinity matrix between modalities. Being competitive, the paper combines of many of the recent advancement, such as Down/Up attention, glimpses and counter model. Down/Up methodology with Co-Attention is new, and beautifully illustrated, showing an exact interaction between question elements and detected-object regions. The final model achieve SOTA results in for a very competitive task.  Clarity/Quality: The paper is well-written. contributions clearly laid out.  Originality: The paper inspired and closely follows work of Lu et. al. [18] parallel model combines with low-rank bilinear pool, such as MCB, MLB, MFH. Significance: The proposed approach demonstrates clear improvements for VQA task. Analysis of different attention methods proves the proposed attention is better than previous work.  Concerns: •Related work of the paper is not accurate. L123-124 claim that Co-Attention neglected interactions between modalities. The parallel version of [18] or [20] defiantly capture the interactions between the modalities via similarity matrix. The parallel version was ignored in Quantitative analysis. I think you should elaborate about the exact differences between your technique, and parallel-like techniques.   •Related work is also not comprehensive enough in my opinion. As attention paper, the discussion is limited only to very similar approaches. For instance, “Structured Attentions for Visual Question Answering” from ICCV17, solves multimodal attention as structure prediction, or ‘’High-Order Attention Models for Visual Question Answering`` from NIPS17, which also discuss Co-Attention, and suggests a network with combined unary and pairwise information (i.e., multiply interactions), which surpass [18], and was not mentioned.  •Fig 2. How exactly have you tested the different attentions? Did you replace only the attention unit? Did you tune the network with different architecture?  • why this model is avoiding overfitting, or able to use more glimpses?  Conclude:  I like the paper, though it feels a bit too concentrated on the competitive aspects, i.e. accuracy score. I am missing a more comprehensive justification and comparison to similar co-attention techniques. The attention unit is novel, but incremental from common co-attention practice. Nevertheless, the overall quality of the submission is good, with SOTA performance and a complete analysis for many design choices, therefore I’m in favor for acceptance.  After author response: I thank the authors for their response. I never intended to misjudge the significance of BAN results vs previous work, I was more confused about the important ideas of BAN versus other approaches that use bilinear attention. While the difference from  iteratively approaches for co-attention is obvious, I'm still confused about the difference from previous work that use bilinear attention. The paper only mention a very early approach by Xu and Saenko. The proposed low rank bilinear operation is not new, and was already suggested in the papers I mentioned.  Arguing that bilinear approaches are weak against iterative approaches does not align with the paper main argument. To progress the research in this direction,  I hope you will chose to clarify the importance of your approach for attention in the revised version, and looking forward to see the paper published at NIPS.