Stacked Semantics-Guided Attention Model for Fine-Grained Zero-Shot Learning
Zero-Shot Learning (ZSL) is generally achieved via aligning the semantic relationships between the visual features and the corresponding class semantic descriptions. However, using the global features to represent fine-grained images may lead to sub-optimal results since they neglect the discriminative differences of local regions. Besides, different regions contain distinct discriminative information. The important regions should contribute more to the prediction. To this end, we propose a novel stacked semantics-guided attention (S 2 GA) model to obtain semantic relevant features by using individual class semantic features to progressively guide the visual features to generate an attention map for weighting the importance of different local regions. Feeding both the integrated visual features and the class semantic features into a multi-class classification architecture, the proposed framework can be trained end-to-end. Extensive experimental results on CUB and NABird datasets show that the proposed approach has a consistent improvement on both fine-grained zero-shot classification and retrieval tasks.
This paper aims at using attention on image regions to improve fine grained zero shot learning task. The proposed method is build up on existing zero shot learning frame work with a novel stacked attention branch, which matches global image and text representation and use it to weight regional features. The proposed method shows better performance in two bird datasets.  Strengths: - The idea to put attention into zero shot learning task, especially first regresses global image feature to textual representation, then use it to guide attention, is new.  - Proposed method shows better performance in two datasets by a significant margin, especially in zero-shot retrieval task.  Weaknesses:  - This paper misses a few details in model design and experiments: A major issue is the "GTA" / "DET" feature representation in Table 1. As stated in section 4.1, image regions are extracted from ground-truth / detection methods. But what is the feature extractor used on top of those image regions? Comparing resnet / densenet extracted features with vgg / googlenet feature is not fair.  - The presentation of this paper can be further improved. E.g. paragraph 2 in intro section is a bit verbose. Also breaking down overly-long sentences into shorter but concise ones will improve fluency.  Some additional comments: - Figure 3: class semantic feature should be labeled as "s" instead of "c"?  - equation 1: how v_G is fused from V_I? please specify.  - equation 5: s is coming from textual representations (attribute / word to vec / PCA'ed TFIDF). It might have positive / negative values? However the first term h(W_{G,S}, v_G) is post ReLU and can only be non-negative?  - line 157: the refined region vector is basically u_i = (1 + attention_weight)  * v_i. since attention weight is in [0, 1] and sums up to 1 for all image regions. this refined vector would only scales most important regions by a factor of two before global pooling? Would having a scaling variable before attention weight help?  -  line 170: class semantic information is [not directly] embedded into the network?   - Equation 11: v_s and u_G are both outputs from trained-network, and they are not normalized? So minimize L-2 loss could be simply reducing the magnitude of both vectors?  - Line 201: the dimensionality of each region is 512: using which feature extractor?   - Section 4.2.2: comparing number of attention layers is a good experiment. Another baseline could be not using Loss_G? So attention is only guided by global feature vector.   - Table 4: what are the visual / textual representations used in each method? otherwise it is unclear whether the end-to-end performance gain is due to proposed attention model.