Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application. In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent code z that specifies the customization to the individual data sequence. This enables style transfer, interpolation and morphing within generated sequences. We show the MTDS can improve predictions via latent code interpolation, and avoid the long-term performance degradation of standard RNN approaches.
This paper proposes a multi-task dynamical system for sequence generation. The model learns a number of parameters that represents the latent code z. The learned model can generate the customized individual data sequence and provide the smooth interpolation in the sequence space. The experiments on the synthetic data and the ocap dataset show the customization is beneficial for prediction tasks and enables for style transfer and morphing within generated sequences. 
The mathematical definition and the proof in this paper are well justified. However, I had a hard time understanding the contribution of this paper. 
- The main motivation of this paper is to treat each sequence as a task in the training set (customization of the individual data sequence). What are the advantages of this setting for sequence generation?
- Separate parameterization of the latent variable z for different tasks seems to be a key idea in this paper. What is the main benefit of parameterizing each latent variable? Does it improve diversity?
- If not, what is the key idea that the proposed model can generate diverse sequences compare to other generative models (e.g., GAN or VAE-based models). 
- The authors claim that the proposed approach provides grater data efficiency. How is it compare to other generative models?
- Overall, it is not clear the benefit of the proposed model over existing algorithms. This paper needs to provide comparisons with any other existing models, especially on Mocap data, 
- What are the differences between pooled models and single-task models?
---- 
I increase the rating after rebuttal, since the idea is interesting and the paper has been improved. However, I still think the experiments and the comparisons are unconvincing.