A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning
This paper takes a step towards temporal reasoning in a dynamically changing video, not in the pixel space that constitutes its frames, but in a latent space that describes the non-linear dynamics of the objects in its world. We introduce the Kalman variational auto-encoder, a framework for unsupervised learning of sequential data that disentangles two latent representations: an object's representation, coming from a recognition model, and a latent state describing its dynamics. As a result, the evolution of the world can be imagined and missing data imputed, both without the need to generate high dimensional frames at each time step. The model is trained end-to-end on videos of a variety of simulated physical systems, and outperforms competing methods in generative and missing data imputation tasks.
The paper presents a time-series model for high dimensional data by combining variational auto-encoder (VAE) with linear Gaussian state space model (LGSSM). The proposed model takes the latent repressentation from VAE as the output of LGSSM. The exact inference of linear Gaussian state space model via Kalman smoothing enables efficient and accurate variational inference for the overall model. To extend the temporal dynamics beyond linear dependency, the authors use a LSTM to parameterize the matrices in LGSSM. The performance of the proposed model is evaluated through bouncing ball and Pendulum experiments.
LGSSM with dynamic parameterization is a nice trick to learn complex temporal dynamics through a simple probabilistic model, but has been exploited in the literature, e.g., [M. Seeger, D. Salinas, V. Flunkert NIPS 2016]. The proposed method does not really disentangle the object representation from the temporal dynamic, as the model is unlikely to infer the correct dynamics of an unseen object. The proposed method only models the temporal dynamic on top of the low dimensional latent representation from VAE.
The proposed method is evaluted on some simple examples, where the object representation is simple (mostly single object with low resolution image) and the dynamic is also simple (bouncing of a ball or Pendulum). It would be more convencing to show experiments with either complex data representation or complex dynamics (e.g., human motion). In Pendulum experiment, the proposed method is compared with other two methods in terms of ELBO. However, it is questionable how to compare ELBO from different models (e.g., the numbers of parameters are probably different.).