Z-Forcing: Training Stochastic Recurrent Networks
Many efforts have been devoted to training generative latent variable models with autoregressive decoders, such as recurrent neural networks (RNN). Stochastic recurrent models have been successful in capturing the variability observed in natural sequential data such as speech. We unify successful ideas from recently proposed architectures into a stochastic recurrent model: each step in the sequence is associated with a latent variable that is used to condition the recurrent dynamics for future steps. Training is performed with amortised variational inference where the approximate posterior is augmented with a RNN that runs backward through the sequence. In addition to maximizing the variational lower bound, we ease training of the latent variables by adding an auxiliary cost which forces them to reconstruct the state of the backward recurrent network. This provides the latent variables with a task-independent objective that enhances the performance of the overall model. We found this strategy to perform better than alternative approaches such as KL annealing. Although being conceptually simple, our model achieves state-of-the-art results on standard speech benchmarks such as TIMIT and Blizzard and competitive performance on sequential MNIST. Finally, we apply our model to language modeling on the IMDB dataset where the auxiliary cost helps in learning interpretable latent variables.
The paper introduces a training technique that encourages autoregressive models based on RNNs to utilize latent variables. More specifically, the training loss of a recurrent VAE with a bidirectional inference network is augmented with an auxiliary loss.
The paper is well structured and written, and it has an adequate review of previous work. 
The technique introduced is heuristic and justified intuitively. The heuristic is backed by empirical results that are good but not ground-breaking. 
The results for speech modelling are very good, and for sequential MNIST good. The auxiliary loss brings no quantitative improvement to text modelling , but an interesting use for the latent variables is shown in section 5.4.
Some concerns:
Line 157 seems to introduce a second auxiliary loss that is not shown in L_{aux} (line 153). Is this another term in L_{aux}? Was the backward RNN pretrained on this objective? More details should be included about this. Otherwise, this paper should be rejected as there is plenty of space left to include the details. 
Line 186: How was the \beta term introduced? What scheduled? Did detail this prove essential? No answers to these questions are provided
Line 191: What type of features were used? Raw signal? Spectral? Mel-spectral? Cepstral? Please specify.
Line 209: Why the change from LSTM to GRU? Was this necessary for obtaining good results? Were experiments run with LSTM?
Line 163: Gradient disconnection here and on paragraph starting on line 127. Can they be shown graphically on Figure 1? Maybe an x on the relevant lines?
Figure 2 has very low quality on a print out. Should be improved.
Line 212: "architecturaly flat" do the refer to number of layers? Do any of the baseline models include information about the 2D structure of the MNIST examples? specify if any does as this would be advantageous for those models.
The section that refers to IWAE: Were the models trained with IWAE or just evaluated? I think it was just evaluated, but it should be more clearly specified. Would it be possible to train using IWAE?