Learning Stochastic Feedforward Neural Networks
Multilayer perceptrons (MLPs) or neural networks are popular models used for nonlinear regression and classification tasks. As regressors, MLPs model the conditional distribution of the predictor variables Y given the input variables X. However, this predictive distribution is assumed to be unimodal (e.g. Gaussian). For tasks involving structured prediction, the conditional distribution should be multi-modal, resulting in one-to-many mappings. By using stochastic hidden variables rather than deterministic ones, Sigmoid Belief Nets (SBNs) can induce a rich multimodal distribution in the output space. However, previously proposed learning algorithms for SBNs are not efficient and unsuitable for modeling real-valued data. In this paper, we propose a stochastic feedforward network with hidden layers composed of both deterministic and stochastic variables. A new Generalized EM training procedure using importance sampling allows us to efficiently learn complicated conditional distributions. Our model achieves superior performance on synthetic and facial expressions datasets compared to conditional Restricted Boltzmann Machines and Mixture Density Networks. In addition, the latent features of our model improves classification and can learn to generate colorful textures of objects.
This paper proposes a neural network architecture that 
      falls somewhere between multilayer perceptrons (MLPs) and sigmoid belief 
      networks (SBNs). The motivation is to permit multimodal predictive 
      distributions (like SBNs) by using stochastic hidden units, but adds 
      deterministic hidden units to smooth the predictive distribution in the 
      case of real-valued data. The paper's main technical contribution is an 
      EM-style algorithm where the E-step uses importance sampling to 
      approximate the posterior and the M-step uses backpropagation to update 
      the parameters. The experiments demonstrate the model's utility on several 
      synthetic and real datasets. Quality: I liked this paper; the use 
      of stochastic and deterministic units seems reasonably justified. The 
      model and inference/learning were well presented and I thought the 
      experiments were thorough. One criticism would be that the datasets 
      considered are either small or subsets of other better-known datasets. In 
      other words, large ML baseline datasets are not considered. However, the 
      paper makes an effort to compare the proposed model against a number of 
      reasonable baselines.Clarity: The paper was generally well 
      organized and well-written. It has a number of small grammatical errors 
      and would benefit from a thorough proofread. Originality: I am not 
      aware of other feedforward neural network architectures that use a mix of 
      deterministic and stochastic units. The inference and learning procedure, 
      while approximate, seem reasonable (and bear a slight resemblance to the 
      sampling methods used in RBM-type models). Significance: The paper 
      is correct in that there is often a need to model multimodal predictive 
      distributions in machine learning. The proposed model is relatively easy 
      to understand and should be able to be easily implemented and extended. I 
      think it would be considered by other practitioners. 
      CommentsFigure 1 presents a model where the first and last 
      layer are wholly deterministic and the inner layers contain some 
      stochastic units. Once the reader gets to the experiments, one sees that 
      is the type of architecture used in the experiments. But the paper does 
      not elaborate on this choice. Is there some significance in choosing a 
      first and last fully-deterministic layer, or is this arbitrary?Is 
      it necessary to use sigmoid units for both the deterministic and 
      stochastic units? Other types of nonlinearities are not mentioned in the 
      paper. Page 2, "We note that we can use any other parameterized 
      distribution ... for the output variables. This is a win compared to the 
      Boltzmann Machine family of models, which require the output distribution 
      to be from the exponential family." Is this totally true? I agree that 
      exponential family Boltzmann machine models are convenient and the norm, 
      but isn't one completely free to define the energy function and interpret 
      the distributions however one chooses?The baseline SBN considered 
      in the experiments is trained in the exact same way as the SFNN, only it 
      has no deterministic units. Have you compared to Sigmoid Belief networks 
      trained in other ways? Is there an advantage to training them with the 
      proposed algorithm designed for SFNNs? That would be an interesting 
      secondary result.The experiments in section 3.1 do not say how the 
      test log-probability was approximated for the C-GRBM (which has an 
      intractable partition function, so log likelihood cannot be computed 
      exactly). The following section says that AIS was used; is this also the 
      case for section 3.1?Figure 5 shows how different hyperparamters 
      affect the evaluation and learning of SFNNs, including the absolute number 
      of stochastic nodes per layer. I'd be interested in seeing the ratio of 
      stochastic to deterministic nodes, rather than the absolute number of 
      stochastic nodes.Minor comments- Footnote of pg 1, \sigma 
      is overloaded (used for both variance e.g. \sima_y^2 and also to mean the 
      sigmoid function)- Inline math immediately after Eq 1, bias_2, bias_1 
      is ugly notation; why not use a lowercase letter for the 
      biases?