Professor Forcing: A New Algorithm for Training Recurrent Networks
The Teacher Forcing algorithm trains recurrent networks by supplying observed sequence values as inputs during training and using the network's own one-stepahead predictions to do multi-step sampling. We introduce the Professor Forcing algorithm, which uses adversarial domain adaptation to encourage the dynamics of the recurrent network to be the same when training the network and when sampling from the network over multiple time steps. We apply Professor Forcing to language modeling, vocal synthesis on raw waveforms, handwriting generation, and image generation. Empirically we find that Professor Forcing acts as a regularizer, improving test likelihood on character level Penn Treebank and sequential MNIST. We also find that the model qualitatively improves samples, especially when sampling for a large number of time steps. This is supported by human evaluation of sample quality. Trade-offs between Professor Forcing and Scheduled Sampling are discussed. We produce T-SNEs showing that Professor Forcing successfully makes the dynamics of the network during training and sampling more similar.
This paper proposes a method for training recurrent neural networks (RNN) in the framework of adversarial training. Since RNNs can be used to generate sequential data, the goal is to optimize the network parameters in such a way that the generated samples are hard to distinguish from real data. This is particularly interesting for RNNs as the classical training criterion only involves the prediction of the next symbol in the sequence. Given a sequence of symbols x_1, ..., x_t, the model is trained so as to output y_t as close to x_{t+1} as possible. Training that way does not provide models that are robust during generation, as a mistake at time t potentially makes the prediction at time t+k totally unreliable.  This idea is somewhat similar to the idea of computing a sentence-wide loss in the context of encode-decoder translation models. The loss can only be computed after a complete sequence has been generated.
The idea in this paper is interesting and well motivated. When training an RNN for generation, using the likelihood of the observed data is not the proper criterion.   Even though the problem and approach are interesting, I find the description of the training objective (Sec. 2) very confusing and unclear. Precise remarks follow: - The random variable y is present in two expectations in Eq. (1). - The function B takes y as an input, but actually does not need to. Given an RNN with a given sequence of inputs, all the y can be computed. Moreover, taking the expectation with respect to y following the distribution parametrized by \theta_g does not make sense. Also, I guess that the authors use an empirical approximation of this expectation. It might be cleared to write it as an average on training samples. - The adversarial training is a min-max problem (Googfellow et al., 2014). The authors state that they "perform SG steps on ..." and "always do gradient steps on ...". This means that the two parameters are optimized using different objective functions. One could probably get the same results by maximizing -log(D) instead of minimizing -log(1-D). This would allow the authors to write the optimization problem they try to solve. - The different terms of the objective are not weighted in any way. It would be interesting to see how a trade-off between NLL and the other terms plays in terms of final performance. If the adversarial objective is seen as a "regularizer" (l. 56), it would be good to use a weighting term.  The experimental evaluation regarding the distribution of hidden states and the t-SNE maps is a bit frustrating. No discussion in the paper is proposed and the only explanations are in the caption of Fig. 3. From what I understood, it simply shows the fact that hidden states diverge when generating sequences, as the input shifts away from the true data distribution. More discussion would be welcome.  As far as the experimental evaluation is concerned, the experiments could be more convincing. For the character-level language modeling, the paper compares models trained with and without the adversarial loss. An improvement potentially due to the regularizing effect is shown (0.02 Bits/character). My concern is that the baseline model is not trained properly. Other papers report much lower performances on Penn Treebank, see for instance a recent workshop paper at ICLR2016: Alternative Structures for Character-Level RNNs, Piotr Bojanowski, Armand Joulin and Tomas Mikolov (Table 1). The model they used was a plain RNN and it achieved a validation BPC of 1.42. When the baseline model is not properly tuned, it is hard to assess the impact of the regularization.  Overall this is an paper tackling an interesting and important problem. However, the presentation is a bit overly-complicated and unclear. Experiments exhibiting improved likelihood (imputed to regularization) on held-out data are not convincing and the third experimental claim (tSNE maps) is not properly discussed in the paper. Of course, we need to keep in mind that properly evaluating the performance of generative models is complicated, and the user study shows statistically significant improvements.