Many real-world sequential decision-making problems can be formulated as optimal control with high-dimensional observations and unknown dynamics. A promising approach is to embed the high-dimensional observations into a lowerdimensional latent representation space, estimate the latent dynamics model, then utilize this model for control in the latent space. An important open question is how to learn a representation that is amenable to existing control algorithms? In this paper, we focus on learning representations for locally-linear control algorithms, such as iterative LQR (iLQR). By formulating and analyzing the representation learning problem from an optimal control perspective, we establish three underlying principles that the learned representation should comprise: 1) accurate prediction in the observation space, 2) consistency between latent and observation space dynamics, and 3) low curvature in the latent space transitions. These principles naturally correspond to a loss function that consists of three terms: prediction, consistency, and curvature (PCC). Crucially, to make PCC tractable, we derive an amortized variational bound for the PCC loss function. Extensive experiments on benchmark domains demonstrate that the new variational-PCC learning algorithm benefits from significantly more stable and reproducible training, and leads to superior control performance. Further ablation studies give support to the importance of all three PCC components for learning a good latent space for control.
This paper considers learning low-dimensional representations from high-dimensional observations for control purposes. The authors extend the E2C framework by introducing the new PCC-Loss function. This new loss function aims to reflect the prediction in the observation space, the consistency between latent and observation dynamics, and the low curvature in the latent dynamics. The low curvature term is used to bias the latent dynamics towards models that can be better approximated as locally linear models. The authors provide theory (error bounds) to justify their proposed PCC-Loss function. Then variational PCC is developed to make the algorithm tractable. The proposed method is evaluated in 5 different simulated tasks and compared with the original E2C method and the RCE method. 
The paper is well-written. 
Pros:
- The idea in this paper is quite original. The three principles used to formulate the loss function provide some new insights.
- The authors have proposed a theory to justify the use of their loss function. The technical quality of this part seems solid.
- Simulations have been used to show that the proposed PCC method outperforms E2C and RCE.
-The paper is well written. 
Cons:
- The tasks in this paper are not that complicated. It is unclear whether the proposed method outperforms other model-based RL methods such as Solar and DSAE for practical robotic applications. More comparisons are needed.
- It is also not that clear why one wants to SOC3 to be close to SOC1 in the first place. It seems the true optimization problem should be posed on the space of the original state s. SOC1 is just a surrogate problem for the original problem.
- There seems to be a gap between the proposed theory  and the algorithm implementation. This makes the theory part less useful.
Overall, I think the idea in this paper is interesting. The authors have made a serious effort in coming up principles for model-based RL control. But at this moment it is not that convincing the proposed method will be the best model-based RL method for practical robotic applications. If the authors can address my comments, I will be willing to increase my score.
Minor Comments:
- It seems that for the task the authors have tested their method, it is not that difficult to directly estimate the state. Am I correct here? Can the authors make a comment on this? How to compare their approach and a more direct control approach using estimation of state s? 
- I have never seen the curvature principle in any control papers. Any control reference on why this is a good principle? It seems that the linearization works well when the control inputs are around the reference points. Does the curvature really matter that much for ILQR to work? 
- How to justify the Markovian assumption on x? Just by observation or there is a more principle way to test this assumption on the buffered images?
====================================================
Post-Rebuttal:
After reading the authors' response, I am changing my score to weak accept. Lemma 4 is nice. I have not seen anything similar to this in the controls literature before. The authors have addressed most of my concerns. I still have a few comments for preparing the final version of this paper. 
1. I still don't see why SOC1 is the "original problem." Yes, it is assumed that the true state cannot be directly observed. But if the observations are Markov eventually, then some estimated version of the states can be obtained, right? I think treating SOC1 as the original problem is one possible way of doing things and clearly the authors have built a principled framework for doing things in this way. But treating SOC1 as the original problem seems not the only way of doing things. I hope the authors can clarify this and do not oversell the proposed approach. 
2. I think it is still worth comparing SOLAR and PCC empirically. This will help the readers to choose algorithms when they need. 
3. The comment on the verification of the Markov assumption is hand-waving.  The authors said "A simple test would be to see if a control algorithm with the Markovian assumption works well with our representation or not." Does this mean that the users will not be able to verify this assumption before using the proposed approach to obtain controllers? It will be helpful if the authors can explain this step for one specific example in details.