DeViSE: A Deep Visual-Semantic Embedding Model
Modern visual recognition systems are often limited in their ability to scale to large numbers of object categories. This limitation is in part due to the increasing difficulty of acquiring sufficient training data in the form of labeled images as the number of object categories grows. One remedy is to leverage data from other sources -such as text data -both to train visual models and to constrain their predictions. In this paper we present a new deep visual-semantic embedding model trained to identify visual objects using both labeled image data as well as semantic information gleaned from unannotated text. We demonstrate that this model matches state-of-the-art performance on the 1000-class ImageNet object recognition challenge while making more semantically reasonable errors, and also show that the semantic information can be exploited to make predictions about tens of thousands of image labels not observed during training. Semantic knowledge improves such zero-shot predictions achieving hit rates of up to 18% across thousands of novel labels never seen by the visual model.
Summary of paper: This computer vision paper uses an 
      unsupervised, neural net based semantic embedding of a Wikipaedia text 
      corpus trained using skip-gram coding to enhance the performance of the 
      Krizhevsky et al deep network [11] that won the 2012 ImageNet large scale 
      visual recognition challenge, particularly for zero-shot learning problems 
      (i.e. previously unseen classes with some similarity to previously seen 
      ones). The two networks are trained separately, then the output layer of 
      [11] is replaced with a linear mapping to the semantic text representation 
      and re-trained on ImageNet 1k using a dot product loss reminiscent of a 
      structured output SVM one. The text representation is not currently 
      re-trained. The model is tested on ImageNet 1k and 21k. With the semantic 
      embedding output it does not quite manage to reproduce the ImageNet 1k 
      flat-class hit rates of the original softmax-output model, but it does 
      better than the original on hierarchical-class hit rates and on previously 
      unseen classes from ImageNet 21k. For unseen classes, the improvements are 
      modest in absolute terms (albeit somewhat larger in relative ones). 
      Review: I think this is an interesting paper that can be accepted. 
      The subject area (scaling to very large numbers of visual classes, 
      combining modalities, deep learning) is clearly topical. This is not the 
      first method to combine visual and textual information, but AFAIK it is 
      the first to tackle this problem at such a large scale (all 21k ImageNet 
      classes), especially using a fully unsupervised text model. The absolute 
      improvements in recognition rates are fairly modest, but this is a 
      challenging area where any improvement is welcome. Further points 
      I have several questions for the authors: - The skip-gram 
      textual model is quite weak in the sense that no prior semantics is built 
      in. It seems plausible that strengthening the semantic cues (e.g. with a 
      WordNet distance based regularizer) would improve the embedding. Keying on 
      Wikipaedia structures might also help, e.g. a "blue shark" might have its 
      own page, with links back to the generic "shark" page. - It would 
      be useful to quantify the improvements available by refining the textual 
      embedding during the joint training phase. - How do adjectives / 
      visual attributes come into this? Many classes have descriptive names, 
      e.g. one might expect a "whitecap shark" to have some body part with a 
      white cap. It is not clear to me that the current embedding model exploits 
      such hints / such factorization in any usable way (e.g. as a "white" basis 
      vector to be mixed in to the "shark" dot product). - What are the 
      flat precision scores for zero-shot DeVISE? - I ask because the 
      hierarchical metric necessarily confuses the issue, especially at such low 
      absolute precisions. E.g. if images of blue sharks are classified mainly 
      as sharks, but not particularly as blue ones, the hierarchical score could 
      still be quite high but one would hesitate to claim that a blue shark 
      classifier has been learned. I suspect that this is happening here, at 
      least to some extent. Note added after rebuttal: Excuse me 
      for my garbled question about flat hit@k precision scores for zero shot 
      DeVISE. I realise that you have already given these. What I meant to ask 
      for was scores for a *hierarchical version* of the flat hit@k metric. Flat 
      hit@k is the familiar "best of k" metric -- only the best of the k guesses 
      counts towards the score. Your hierarchical precision@k is interesting but 
      very different: it essentially measures how well the full set of k guesses 
      reproduces the local categorical neighbourhood of the one true class. As 
      such it is less robust than a "best of k" metric: even if a method 
      invariably determines the true class with one of its k guesses, it will 
      score badly on hierarchical precision@k if its other k-1 guesses are 
      typically far from the true class. Given that you report the two metrics 
      face to face and don't fully explain hierarchical precision@k in the 
      paper, unwary readers are likely to think that hierarchical precision@k is 
      a hierarchical "best of k" metric, and hence be mislead by the results. 
      Also, to get a handle on the types of errors that are being made, it would 
      be convenient to report at least some results in a metric that supports a 
      "hops from ground truth" score decompositon. E.g. with the hierarchical 
      best of k metric that counts how many hops from the true class the best 
      [*] of the k guesses is, you can report accuracies as (say) 10% hits at 
      zero hops (the flat hit@k score) plus 5% hits at one hop plus ... If 
      desired this can be refined by breaking hop ties, e.g. counting children 
      as closer than parents you might get "plus 3% hits to a child class plus 
      2% hits to the parent class plus...". [*] Only one of the k 
      guesses ever counts. Ties are broken arbitrarily. Further points: 
      - Please use a uniform reporting convention for all performance 
      score tables -- convert either Table 3 to percentages, or Table 2 to 
      probabilities.