Deep Fisher Networks for Large-Scale Image Classification
As massively parallel computations have become broadly available with modern GPUs, deep architectures trained on very large datasets have risen in popularity. Discriminatively trained convolutional neural networks, in particular, were recently shown to yield state-of-the-art performance in challenging image classification benchmarks such as ImageNet. However, elements of these architectures are similar to standard hand-crafted representations used in computer vision. In this paper, we explore the extent of this analogy, proposing a version of the stateof-the-art Fisher vector image encoding that can be stacked in multiple layers. This architecture significantly improves on standard Fisher vectors, and obtains competitive results with deep convolutional networks at a smaller computational learning cost. Our hybrid architecture allows us to assess how the performance of a conventional hand-crafted image classification pipeline changes with increased depth. We also show that convolutional networks and Fisher vector encodings are complementary in the sense that their combination further improves the accuracy.
This paper uses Fisher Vectors as inner building 
      blocks in a recognition architecture. The basic Fisher vector module had 
      previously demonstrated superior performance in recognition application. 
      Here, it is augmented with discriminative linear projection for 
      dimensionality reduction, and multiscale local pooling, to make it 
      suitable for stacking. Inputs of all layers are jointly used for 
      classification. Experiments show convincing performance on ImageNet. 
      This paper is well organized, well written, and shows good results, 
      establishing that (1) adding another layer of Fisher vector extraction is 
      useful, (2) using both layers jointly is better than using any one of them 
      in isolation (this confirms earlier work, as stated in the paper), (3) 
      using multiscale pooling is better than using a single pooling 
      neighborhood size. Concerning this last point: the idea of multiple 
      pooling scales vs. a single one is analogous to the difference between 
      spatial pyramid and simple bag of words, so this is not really new, and 
      should be mentioned. A small gripe with the writing is with two ways 
      that the paper is sold: (1) the conclusion reads, "the architecture is 
      largely hand-crafted, and requires lighter training." Why is using 
      hand-crafted features seen as a good thing? The fact that other deep 
      networks incorporate training allows adaptation to other domains. For a 
      vision application, training could be avoided by plugging in previously 
      trained weights, using training samples as weights, using only 
      unsupervised pre-training, etc. -- the option to fully train a network 
      simply results in the best performance, but the amount of needed training 
      is flexible if one is willing to accept traning-performance trade-offs. 
      (2) the introduction and conclusion take aim at a strawman: e.g. in 
      the introduction, l.48: "are the deep architectures used in CNN really 
      necessary, or can comparable performance be obtained simply by stacking 
      the well crafted features developed in the computer vision literature?". 
      This might unnecessarily alienate the deep learning crowd, given that it 
      has so often been emphasized that all types of modules are suitable for 
      embedding in deep learning architectures, and bits and pieces of computer 
      vision architectures have in fact been used (e.g. see work by Weston and 
      Collobert; the angle taken by Coates et al. in ref. 22; the use of SVM and 
      spatial pyramids mixed with convolutional network layers in Kavukcuoglu et 
      al. NIPS 2010.) Instead, a better way to describe the contribution of the 
      paper could be to reverse the framing and ask whether conventional 
      computer vision features could benefit from the stacking principles used 
      in deep architectures. Minor comments: Ref. 7 used for instances of 
      sparse coding and max pooling should be replaced by earlier work, e.g.: 
      Yang et al. CVPR 2009, Linear spatial pyramid matching using 
      sparse coding for image classification