Discriminative Transfer Learning with Tree-based Priors
High capacity classifiers, such as deep neural networks, often struggle on classes that have very few training examples. We propose a method for improving classification performance for such classes by discovering similar classes and transferring knowledge among them. Our method learns to organize the classes into a tree hierarchy. This tree structure imposes a prior over the classifier's parameters. We show that the performance of deep neural networks can be improved by applying these priors to the weights in the last layer. Our method combines the strength of discriminatively trained deep neural networks, which typically require large amounts of training data, with tree-based priors, making deep neural networks work well on infrequent classes as well. We also propose an algorithm for learning the underlying tree structure. Starting from an initial pre-specified tree, this algorithm modifies the tree to make it more pertinent to the task being solved, for example, removing semantic relationships in favour of visual ones for an image classification task. Our method achieves state-of-the-art classification results on the CIFAR-100 image data set and the MIR Flickr image-text data set.
The paper presents a multi-task learning approach 
      based on hierarchical Bayesian priors. The priors are combined with a deep 
      NN resulting in a discriminative learning process which encourages sharing 
      information among tasks taking into account different levels of 
      relatedness. The paper is written well and easy to follow. The 
      topic of multi-task learning is a very popular and important topic and the 
      sub issue of considering different level of relatedness has also gained 
      much attention lately. I liked the proposed integration with NN and the 
      good experimental results. I feel the paper does not give enough 
      background on related approaches. There are many related papers, but 
      specifically I feel a discussion on similar hierarchical Bayesian 
      approaches is lacking: Bayesian Multitask Learning with Latent 
      Hierarchies, H. Daume III where a similar prior is seemed to be 
      used for what they denote the domain adaptation approach. Further 
      related work is: Multi-task learning for classification with 
      Dirichlet process priors, Xue et al. 2007 and more papers 
      presenting discriminative approaches dealing with different levels of 
      relatedness: Learning with whom to share in multi-task feature 
      learning, kang et al. Hierarchical regularization cascade for 
      joint learning, zweig et al. Tree-guided group lasso for 
      multi-task regression with structured sparsity. Kim et al. I 
      also have a few questions regarding the algorithm for discovering the 
      hierarchical structure: - The hierarchy discovery is described in 
      the context of the CRP prior but then it is said that solving the 
      inference problem directly is too complex and thus an approximation is 
      proposed. How does this approximation relate to the original prior? I 
      couldn't find the explicit use of the prior (over the generation of a new 
      super class) in the presented approach. - Unless I missed 
      something, it seems that the current presentation and experiments deals 
      only with a two level hierarchy, super class and original classes. Can the 
      approximation method be at all extended to more levels? - The 
      method requires a validation set in order to decide on the best model at 
      each step of the hierarchy construction algorithm. Can we assume a 
      validation set in small sample scenarios (which are really the motivating 
      scenario for multi-task learning)? I saw in the supplementary material 
      this issue is addressed, but didnt understand if the validation set 
      discussed there is the validation set for the hyper-parameters or the 
      hierarchy discovery or both. - How is the convergence of the 
      hierarchical structure measured? - The algorithm assumes an 
      initial good guess. What can be done when such a guess is not available or 
      too costly? would a random initialization work well? It would be nice to 
      see an experiment with a random initialization Generally I 
      liked the experimental section and found it sufficient. My only concern is 
      that I would have preferred to see the experiment on small sample of only 
      a single class done on more examples, then only the dolphin class.