Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images
We introduce Embed to Control (E2C), a method for model learning and control of non-linear dynamical systems from raw pixel images. E2C consists of a deep generative model, belonging to the family of variational autoencoders, that learns to generate image trajectories from a latent space in which the dynamics is constrained to be locally linear. Our model is derived directly from an optimal control formulation in latent space, supports long-term prediction of image sequences and exhibits strong performance on a variety of complex control problems.
This paper proposes a combination of deep learning and stochastic optimal control for control of non-linear dynamical systems from image inputs. The system learns a latent space representation of the true system state based on images of the state using deep variational auto encoders. Additionally, a locally-linear dynamics model is learned on the latent space which is used to generate state space trajectories by applying stochastic optimal control techniques directly on the latent space. The system is tested on four control tasks and the performance is compared with multiple baselines showing good performance.Quality: --------- The paper is logical and sound. The use of (deep) variational encoders to learn the latent space representation mirrors previous work. Enforcing local-linearity on the transition dynamics coupled with the constraint between the transition and inference distributions is crucial for good performance. It would be interesting to see if the authors can model longer term dependencies using RNNs and/or LSTMs. The experiments highlight the strength of the approach, especially the fact that the learned representations are conducive for control. More experiments on higher dimensional (state/control dimensionality) tasks would help. The authors could compare their learned latent representations with those learned by Predictive State Representation based methods.Clarity: --------- The paper is very well written and is easily understandable. The fact that the authors give details on the structure of their networks and various hyper-parameters in the supplementary materials is appreciated. In Sec 2.2, line 103, correct c_T(x_t,u_t) = c(x_T, u_T).Originality: ------------- The main novelty of the paper lies in controlling the system directly on the latent space and the restrictions on the latent transition dynamics. This makes the method more stable and improves performance significantly as compared to prior approaches.  Significance: --------------- The paper advances the state of the art in pixel based control of non-linear dynamical systems and is very relevant to the deep learning, RL and representation learning community. Related citation to add: Boots et al., Closing the Learning-Planning Loop with Predictive State Representations, RSS 2010 -> The authors learn a model of the environment from camera data and plan on the learned modelI vote for a clear accept.