Top-Down Regularization of Deep Belief Networks
Designing a principled and effective algorithm for learning deep architectures is a challenging problem. The current approach involves two training phases: a fully unsupervised learning followed by a strongly discriminative optimization. We suggest a deep learning strategy that bridges the gap between the two phases, resulting in a three-phase learning procedure. We propose to implement the scheme using a method to regularize deep belief networks with top-down information. The network is constructed from building blocks of restricted Boltzmann machines learned by combining bottom-up and top-down sampled signals. A global optimization procedure that merges samples from a forward bottom-up pass and a top-down pass is used. Experiments on the MNIST dataset show improvements over the existing algorithms for deep belief networks. Object recognition results on the Caltech-101 dataset also yield competitive results.
The key idea of this paper is to use the latent vector 
      generated from upper layers of a Deep Belief Network as target vectors, by 
      adding a regularization cross-entropy part to the standard unsupervised 
      training loss, so as to encourage reconstructions from bottom and top to 
      match. Instead of the standard two-stage training of deep architectures 
      (unsupervised layer-by-layer pretraining, then full network supervised 
      training), training is conducted in three stages with an intermediate 
      stage that uses this hybrid loss. Experimental results show that the 
      intermediate stage substantially improves results on MNIST and Caltech101. 
      This regularization of intermediate layers by top-down generative 
      weights shows good results; the paper is clear and shows how the proposed 
      intuitive way of bridging unsupervised and supervised training indeed 
      improves performance. The paper is overall rather clear although 
      there are some problems (see below). Organization: the paper flows 
      well. One minor comment: Section 3: it would be easier for the reader 
      to get the key idea if the intuition was given briefly at the very 
      beginning of section 3, instead of waiting until line 192 -- in the 
      current form, the reader is first left wondering what will be used as 
      target vector. E.g., before the paragraph "Generic Cross-Entropy 
      Regularization", insert a one-sentence summary saying that the main 
      intuition is to use latent vector generated from upper layers as target 
      vectors. Questions/comments: Line 044: Error backpropagation is a 
      way to propagate the error across the lower layers of the networks; it 
      does not say anything about being supervised or discriminative. Indeed one 
      can very well backpropagate unsupervised reconstruction errors across 
      layers. What makes the last training phase supervised is merely *the 
      choice of the criterion* used for training, whose error is then 
      backpropagated. Same goes for Line 290. L.104: using the softmax in 
      Eq. 8 results in the units *not* being independent, and does not 
      correspond to the energy given in Eq.6: this is not what is depicted in 
      Fig.2 but corresponds to a top logistic regression layer. Fig.2 and Eq.6 
      correspond to a logistic just as in Eq.4, with v and d instead of w 
      and c, NOT the softmax equation given in Eq.8. Eq. 6 has exactly the same 
      form for y and x so the resulting probability forms should not differ. 
      L. 262: I am a bit confused: if y is obtained by softmax, as stated 
      l.261, is the label never used? It seems that instead the last layer z_L 
      should be obtained by softmax (or independent binary activation, see 
      comment above -- not sure what was used), and mixed with plugged in ground 
      truth labels y during training. Is this what was done? Minor comments: 
      writing: use more articles (e.g. "A RBM is *a* bipartite Markov random 
      field," "This enables *the* model to be sampled") L.44: "Given a set 
      of trainig data, the unsupervised learning phase initializes the 
      parameters in only one way, regardless of the ultimate classification 
      task." This is confusing (besides, it unnecessarily narrows the context to 
      classification as the final task). If what is meant is that the 
      unsupervised criterion is agnostic to the final task of interest, then a 
      straightforward formulation is better, e.g.: "The unsupervised learning 
      phase does not take into account the ultimate task of interest, e.g. 
      classification." Line 052- "same gradient descent convention": 
      gradient descent is not a convention. "procedure" better. Line 090: 
      missing J: V \in \R^{J \times C}. Line 112: Bengio, not Benjio. 
      Line 270-278: confusing. Change presentation. Line 344: the best 
      score is obtained with setting 3, not 2.