STCN: STOCHASTIC TEMPORAL CONVOLUTIONAL NETWORKS
Convolutional architectures have recently been shown to be competitive on many sequence modelling tasks when compared to the de-facto standard of recurrent neural networks (RNNs), while providing computational and modeling advantages due to inherent parallelism. However, currently there remains a performance gap to more expressive stochastic RNN variants, especially those with several layers of dependent random variables. In this work, we propose stochastic temporal convolutional networks (STCNs), a novel architecture that combines the computational advantages of temporal convolutional networks (TCN) with the representational power and robustness of stochastic latent spaces. In particular, we propose a hierarchy of stochastic latent variables that captures temporal dependencies at different time-scales. The architecture is modular and flexible due to decoupling of deterministic and stochastic layers. We show that the proposed architecture achieves state of the art log-likelihoods across several tasks. Finally, the model is capable of predicting high-quality synthetic samples over a long-range temporal horizon in modeling of handwritten text.
This paper presents a generative sequence model based on the dilated CNN
popularized in models such as WaveNet. Inference is done via a hierarchical
variational approach based on the Variational Autoencoder (VAE). While VAE
approach has previously been applied to sequence modeling (I believe the
earliest being the VRNN of Chung et al (2015)), the innovation where is the
integration of a causal, dilated CNN in place of the more typical recurrent
neural network. 
The potential advantages of the use of the CNN in place of
RNN is (1) faster training (through exploitation of parallel computing across
time-steps), and (2) potentially (arguably) better model performance. This
second point is argued from the empirical results shown in the
literature. The disadvantage of the CNN approach presented here is that
these models still need to generate one sample at a time and since they are
typically much deeper than the RNNs, sample generation can be quite a bit
slower.
Novelty / Impact: This paper takes an existing model architecture (the
causal, dilated CNN) and applies it in the context of a variational
approach to sequence modeling. It's not clear to me that there are any
significant challenges that the authors overcame in reaching the proposed
method. That said, it certainly useful for the community to know how the
model performs.
Writing: Overall the writing is fairly good though I felt that the model
description could be made more clear by some streamlining -- with a single
pass through the generative model, inference model and learning. 
Experiments: The experiments demonstrate some evidence of the superiority
of this model structure over existing causal, RNN-based models. One point
that can be drawn from the results is that a dense architecture that uses multiple levels of the
latent variable hierarchy directly to compute the data likelihood is
quite effective. This observation doesn't really bear on the central message
of the paper regarding the use of causal, dilated CNNs. 
The evidence lower-bound of the STCN-dense model on MNIST is so good (low)
that it is rather suspicious. There are many ways to get a deceptively good
result in this task, and I wonder if all due care what taken. In
particular, was the binarization of the MNIST training samples fixed in
advance (as is standard) or were they re-binarized throughout training? 
Detailed comments:
- The authors state "In contrast to related architectures (e.g. (Gulrajani et
al, 2016; Sonderby et al. 2016)), the latent variables at the upper layers
capture information at long-range time scales" I believe that this is
incorrect in that the model proposed in at least Gulrajani et al also 
- It also seems that there is an error in Figure 1 (left). I don't think
there should be an arrow between z^{2}_{t,q} and z^{1}_{t,p}. The presence
of this link implies that the prior at time t would depend -- through
higher layers -- on the observation at t. This would no longer be a prior
at that point. By extension you would also have a chain of dependencies
from future observations to past observations. It seems like this issue is
isolated to this figure as the equations and the model descriptions are
consistent with an interpretation of the model without this arrow (and
including an arrow between z^{2}_{t,p} and z^{1}_{t,p}.
- The term "kla" appears in table 1, but it seems that it is otherwise not
defined. I think this is the same term and meaning that appears in Goyal et
al. (2017), but it should obviously be defined here.