iclr2020_LLQL
Deep reinforcement learning (RL) algorithms can learn complex policies to optimize agent operation over time. RL algorithms have shown promising results in solving complicated problems in recent years. However, their application on real-world physical systems remains limited. Despite the advancements in RL algorithms, the industries often prefer traditional control strategies. Traditional methods are simple, computationally efficient and easy to adjust. In this paper, we propose a new Q-learning algorithm for continuous action space, which can bridge the control and RL algorithms and bring us the best of both worlds. Our method can learn complex policies to achieve long-term goals and at the same time it can be easily adjusted to address short-term requirements without retraining. We achieve this by modeling both short-term and long-term prediction models. The short-term prediction model represents the estimation of the system dynamic while the long-term prediction model represents the Q-value. The case studies demonstrate that our proposed method can achieve short-term and long-term goals without complex reward functions.
This paper proposes a Locally Linear Q-Learning (LLQL) method for continuous action control. It uses a short-term prediction model and a long-term prediction model to generate actions that achieve short-term and long-term goals simultaneously. The problem the paper seeks to solve is important. However, this paper has several issues.
First, there seems to be an over-claim of the contribution. The proposed method is more like hybrid of model-based and model-free RL method. Specifically, the short-term prediction model is in fact the linearized dynamic system with system parameters modeled by deep neural networks, while the long-term prediction model is in fact different (state- and action-) value functions. For this reason, it is probably unnecessary to name them as “short-term network” or “long-term network”, since they are simply the system model (or the model-based part) and the value functions (the model-free part).
Second, the proposed method is not sufficiently evaluated. It is only evaluated on the toy Mountain Car task (and the Crane system in supplementary material). In order to justify the performance of an RL algorithm for continuous action space, it should be at least evaluated on the set of MuJoCo tasks. 
Third, the proposed method is not sufficiently compared with different baselines. In Figures 3-5, the proposed LLQL algorithm is never compared to any baseline method, leaving it open whether it is actually better than earlier methods like DDPG. In Table 1, LLQL is compared to DDPG (a model-free method), and is shown to achieve better performance. However, this seems to be unfair because the proposed method is in fact a model-based RL algorithm. Therefore, it should at least compare to other model-based algorithms (and also other riche set of safe-exploration RL methods).
Other comments:
•	In eqn. (6), \gamma should be \gamma^{i-k}?
•	In the paragraph after (8), “Q-learning algorithms (16)…” is referring to a wrong equation (16) for Q-learning. Or probably the authors are not using the correct format to cite the reference. (This seems to happen repeatedly in later part of the paper such as as in the paragraph between (14) and (15).) It confuses the equation number and the reference number.
•	More explanation should be given about d(x_k|\theta^d) and h(x_k|\theta^h) after (15). The meaning of them has never to defined before.