A 2 -Nets: Double Attention Networks
Learning to capture long-range relations is fundamental to image/video recognition. Existing CNN models generally rely on increasing depth to model such relations which is highly inefficient. In this work, we propose the "double attention block", a novel component that aggregates and propagates informative global features from the entire spatio-temporal space of input images/videos, enabling subsequent convolution layers to access features from the entire space efficiently. The component is designed with a double attention mechanism in two steps, where the first step gathers features from the entire space into a compact set through second-order attention pooling and the second step adaptively selects and distributes features to each location via another attention. The proposed double attention block is easy to adopt and can be plugged into existing deep neural networks conveniently. We conduct extensive ablation studies and experiments on both image and video recognition tasks for evaluating its performance. On the image recognition task, a ResNet-50 equipped with our double attention blocks outperforms a much larger ResNet-152 architecture on ImageNet-1k dataset with over 40% less the number of parameters and less FLOPs. On the action recognition task, our proposed model achieves the state-of-the-art results on the Kinetics and UCF-101 datasets with significantly higher efficiency than recent works.
This paper proposes the “double attention block” to aggregate and propagate informative global features from the entire spatio/spatio-temporal space of input. Specifically, the model first generates a set of attention distributions over the input and obtains a set of global feature vectors based on the attention. Then, for each input position, it generates another attention distribution over the set of global feature vectors and uses this to aggregate those global feature vectors into a position-specific feature vector. The proposed component can be easily plugged into existing architectures. Experiments on image recognition (ImageNet-1k) and video classification (Kinetics, UCF-101) show that the proposed model outperforms the baselines and is more efficient.  Strength 1.    The proposed module is efficient and easy to adapt to modern frameworks. 2.    Extensive empirical experiments show that the proposed method outperforms consistently outperform other baselines.  Weakness 1.    I am concerned about the originality of the proposed model. The idea of applying an attention over the global feature vectors is conceptually similar to the Transformer architecture (Vaswani et al.). 2.    Lack of theoretical analysis of the proposed model. 3.    The improvement over the baselines is not significant. For example, in the ablation studies (Table 2 and 3), the relative error reduction over “Non-local neural networks” is less than 2%.