Zero-Shot Learning Through Cross-Modal Transfer
This work introduces a model that can recognize objects in images even if no training data is available for the object class. The only necessary knowledge about unseen visual categories comes from unsupervised text corpora. Unlike previous zero-shot learning models, which can only differentiate between unseen classes, our model can operate on a mixture of seen and unseen classes, simultaneously obtaining state of the art performance on classes with thousands of training images and reasonable performance on unseen classes. This is achieved by seeing the distributions of words in texts as a semantic space for understanding what objects look like. Our deep learning model does not require any manually defined semantic or visual features for either words or images. Images are mapped to be close to semantic word vectors corresponding to their classes, and the resulting image embeddings can be used to distinguish whether an image is of a seen or unseen class. We then use novelty detection methods to differentiate unseen classes from seen classes. We demonstrate two novelty detection strategies; the first gives high accuracy on unseen classes, while the second is conservative in its prediction of novelty and keeps the seen classes' accuracy high.
This paper proposes a model for labeling images with 
      classes for which no example appear in the training set, which is based on 
      a combination of word and image embeddings and novelty detection. Using 
      distances in th embedding space between test images and unseen and seen 
      class labels, the approach is able to assign a probability for a new image 
      to be from an unseen class. This is later used to decide which classifier 
      to use (one designed for seen classes the other for unknown ones). Results 
      on CIFAR10 are provided. Clarity: The paper is clearly 
      written. Its motivation and studied task are interesting and ambitious. 
      Originality: There is not much technical novelty since image and 
      word embedding methods come from previous work and so do novelty 
      detections models (with a slight modification of [Kriegel et al. CIKM09]). 
      Still, the combination of those approach appears to be brand new and is 
      not trivial. Significance: The importance of the problem 
      considered here make that this paper could have quite an impact. However, 
      the true application setting of such work (large numbers of classes, 
      imbalance) is still quite far from what is proposed in this paper, which 
      is yet an interesting step towards it. Indeed, all experiments here 
      considers balanced classes, and it seems that such embedding-based work 
      might suffer with imbalance (and the isometric Gaussian assumption), 
      especially when the majority are examples from unseen classes. Similarly, 
      this system needs "well-chosen" unseen classes to perform well (as shown 
      in Fig. 3, where the system performs as chance when unseen classes are cat 
      and dog). Still, these difficulties are shared by any existing system that 
      I know of. Quality: The overall quality of the work is good but 
      some questions remain. The more important point is in Section 6.3, where 
      the full pipeline is used. This shows that in this setting, the Gaussian 
      model outperforms the more sophisticated LoOP approach. Could it be due to 
      the cutoff fraction for log probabilities, which is well tuned for the 
      Gaussian model? How has it been chosen? On the training set? The LoOP does 
      not require (nor benefit from) such tuning. What is the proportion of test 
      examples set as unseen and seen in this combined setting? In 
      Section 6.6, it is not specified if LoOP or the Gaussian model was used. 
      According to the shape of curve of Fig. 5, I would say the Gaussian model. 
      Is it true? Does it mean that this simpler approach is actually the one to 
      choose for good performances with increased number of classes? 
      Other remarks/questions: - Are the word vectors $w_y$ updated? 
      - There seems to be an error in the equation defining lof(.): in the 
      expectation (denominator), shouldn't we read pdist_\lambda(q,C(q)) ? (q 
      instead of f). - A legend should be added for Figure 4 (left and 
      center) indicating which curve correspond to accuracy on seen class, and 
      to unseen. - Comparison with (Lampert et al., 09) in Section 6.3 is 
      interesting. One should also compare with "What Helps Where -- And Why? 
      Semantic Relatedness for Knowledge Transfer" by Rohrbach et al. CVPR 2010, 
      which improved over Lampert and proposes an approach that does not rely on 
      attributes for zero-shot learning. - The results with random 
      distractor noise should be added in Figure 6. After rebuttal: I 
      read the rebuttal which answered conveniently some questions I had. But 
      does not completely justify the novelty.