We address the problem of learning fine-grained cross-modal representations. We propose an instance-based deep metric learning approach in joint visual and textual space. The key novelty of this paper is that it shows that using per-image semantic supervision leads to substantial improvement in zero-shot performance over using class-only supervision. On top of that, we provide a probabilistic justification for a metric rescaling approach that solves a very common problem in the generalized zero-shot learning setting, i.e., classifying test images from unseen classes as one of the classes seen during training. We evaluate our approach on two finegrained zero-shot learning datasets: CUB and FLOWERS. We find that on the generalized zero-shot classification task CLAREL consistently outperforms the existing approaches on both datasets.
The paper proposes to to use four different losses to train a joint text-image embedding space for zero-shot learning. The four losses consist of a classification loss given text descriptions, a classification loss given images, two contrastive losses given pairs of text and images. The paper also discusses how to balance seen and unseen classes, and it seems that, empirically, embeddings for seen classes are closer together while the embeddings for the unseen classes are further apart. A scaling factor that makes sure these distances are comparable for seen and unseen classes is introduced, and the paper gives an explanation for such a scaling. The final performance on the CUB and FLOWERS data set is impressive.
I am giving a score of 3. The engineering effort in the paper is appreciated, but the novelty is lacking. The losses introduced in the paper, broadly classified as contrastive loss, has been around since (Frome et al., 2013). See also (Karpathy et al., 2014; Wang et al., 2016) and in particular (Xian et al., 2016). The subtle differences between all these losses are whether there is a margin, whether the loss is smooth (i.e., a softmax as opposed a max), and how the negative samples are selected. I am surprised that the loss function being used by many are considered a contribution in the paper. The property of metric scaling is certainly interesting, but it does not answer why metric scaling is needed in the first place. Overall, the novelty is limited.
Below are some minor points and questions for the paper.
... anchor embeddings learned in one modality as prototypes ...
--> the word "prototype" is used extensively in the rest of the paper. i know this is a common term in the zero-shot learning community, but it might be good to give a formal definition early or at least give an informal definition.
Algorithm 1
--> what is the choice of d in the experiments?
... the probability of image v_i and text t_j to belong to the same object distance
--> isn't this the constrastive loss? what does it mean for the two to belong to the same object distance?
equations (2)
--> it seems that v_i and t_i are paired, but it is unclear from the text.
equation (1) and (2)
--> have you considered other (distributions of) negative samples?
... P{\hat{y} \in Y^{tr} | y_v \in Y^{ts}} is significantly greater than P{\hat{y} \in Y^{tr} | y_v \in Y^{tr}}
--> what's the definition of P{\hat{y} | y_v}? the notation is confusing and hinders the understanding of the rest of the discussion. it's probably easier to just talk about distances and use the nearest neighbor argument.
Figure 2
--> what's different between the 10 different runs? random seeds?
Figure 3
--> why is the performance significantly lower when lambda=0? can this be related to how negative samples are selected?
References:
DeViSE: a deep visual-semantic embedding model
A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. Ranzato, T. Mikolov
Neurips, 2013
Deep fragment embeddings for bidirectional image sentence mapping
A. Karpathy, A. Joulin, F. Li
Neurips, 2014
Learning deep structure-preserving image-text embeddings
L. Wang, Y. Li, S. Lazebnik
CVPR, 2016
Latent embeddings for zero-shot classification
Y. Xian, Z. Akata, G. Sharma, Q. Nguyen, M. Hein, B. Schiele
CVPR, 2016