Multimodal Residual Learning for Visual QA
Deep neural networks continue to advance the state-of-the-art of image recognition tasks with various methods. However, applications of these methods to multimodality remain limited. We present Multimodal Residual Networks (MRN) for the multimodal residual learning of visual question-answering, which extends the idea of the deep residual learning. Unlike the deep residual learning, MRN effectively learns the joint representation from vision and language information. The main idea is to use element-wise multiplication for the joint residual mappings exploiting the residual learning of the attentional models in recent studies. Various alternative models introduced by multimodality are explored based on our study. We achieve the state-of-the-art results on the Visual QA dataset for both Open-Ended and Multiple-Choice tasks. Moreover, we introduce a novel method to visualize the attention effect of the joint representations for each learning block using back-propagation algorithm, even though the visual features are collapsed without spatial information.
The authors proposed a new network architecture to extend the deep residual learning to multi-modal inputs, applied to a visual question-answering task.  Unlike some previous work where explicit attention network mechanism was utilized, the proposed method follows a previous work using element-wise multiplication between question and visual feature vectors for the joint residual function representation, seen as an implicit attention mechanism. The authors experimented with multiple variants of the architecture and settled on a relatively simple learning block in a 3-block layered network. The combination of deep residual learning and implicit attention were shown to be effective on the Visual QA data set and out-performed other state-of-the-art results. To visualize the attention effect of multi-modal residual learning, the authors proposed a technique to display the difference between visual input and the joint residual mapping with back-propagation for each learning block. Examples from visual QA task intuitively demonstrated the implicit attention effect.
The authors successfully built upon two effective ideas, the deep residual learning and element-wise multiplication for implicit attention, and created a solution for general multi-modal tasks. Experiments were carefully run to select an optimal architecture and hyper-parameters for the targeted Visual QA task.  The results appeared to be superb, compared to previous studies with various deep learning techniques. It would be helpful if the authors can present additional comparison with existing techniques in terms of model parameter size, as well as amount of data required for learning.  It would also be interesting to separately assess the value of residual learning and implicit attention on the Visual QA task, to help understand which aspect is the most critical. The proposed visualization method was intuitively appealing and the examples demonstrated its effectiveness in explaining the implicit attention mechanism. It would be helpful if the authors can include additional explanation on the differences between the three images at each block layer, and perhaps provide an intuitive explanation why three layer appears to be optimal for this task.