Prediction of future states of the environment and interacting agents is a key competence required for autonomous agents to operate successfully in the real world. Prior work for structured sequence prediction based on latent variable models imposes priors with limited expressiveness or are difficult to optimize e.g. determining the number of Gaussian mixture components which makes it challenging to fully capture the multi-modality of the distribution of the future states. In this work, we introduce Conditional Flow Variational Autoencoders (CF-VAE) using our novel conditional normalizing flow based prior to capture complex multi-modal conditional distributions for effective structured sequence prediction. Moreover, we propose two novel regularization schemes which stabilizes training and deals with posterior collapse for stable training and better fit to the target data distribution. Our experiments on three multi-modal structured sequence prediction datasets -MNIST Sequences, Stanford Drone and HighD -show that the proposed method obtains state of art results across different evaluation metrics.
The paper proposes a combination of conditional VAE wtih normalising flows priors and posterior regularisation strategies to capture the diversity of multi-modal trajectories of complex motion patterns. The paper argues that more flexible priors over the latent space can provide posteriors that more closely resemble the trajectories observed in the training data. To this end, the paper presents a derivation of the evidence lower bound for VAEs with normalising flows and discusses the effect of fixing the variance of the posterior to reduce instability during training. Additionally, it shows that conditioning the regularisation on whether or not the dataset contains a dominating mode leads to more diversity and captures minor modes more effectively. Experiments are reported on sequence datasets of handwritten digits, and two datasets with trajectories of vehicles in traffic. 
A central point the paper makes is the importance of prior distributions for the latent space in VAEs such that it can capture diverse modes of trajectories. It is well known that more flexible priors such as MoG lead to better generative power as shown in Tomczak and Welling, 2018. The paper focuses on an extension of the work by Ziegler and Rush, 2019 which proposes normalising flows as priors to capture sequences, conditioned on the initial part of the trajectory. This extension is relatively simple, but does address the specifics of the problem well. 
In general, the paper is well written and clear. The main innovation, in my opinion, is the combination of several ideas applied to the problem of sequence prediction. While none of the ideas, in isolation, are significantly new, the combination can be useful to this particular problem. However, I would like feedback from the authors on the following two main points below which are the main weaknesses of the paper:
1) Posterior regularisation: The posterior regularisation strategies, while intuitive, are very ad-hoc and somewhat contrary to the Bayesian framework. It is difficult to see how the variance in pR and the "dominant mode" detector in cR can be estimated automatically. Within a Bayesian framework it would be much more natural to place a prior distribution over the variance and marginalise it out within the variational inference procedure. For the other regularisation (cR), how is the dominant mode detected? 
2) Experiments: A major concern reported throughout the paper is the instability of training and the risk for overfitting. I do not think the experiments demonstrate how stable and robust the method is to different initialisations, seeds, training data shuffles, etc. I strongly suggest the authors to run cross validation experiments and report the mean and standard deviation for all methods being compared. Also, how sensitive are the results to different values of C? How to decide whether to use cR or not when don't have access to the ground truth?