Reinforcement Learning (RL) problem can be solved in two different ways -the Value function-based approach and the policy optimization-based approach -to eventually arrive at an optimal policy for the given environment. One of the recent breakthroughs in reinforcement learning is the use of deep neural networks as function approximators to approximate the value function or q-function in a reinforcement learning scheme. This has led to results with agents automatically learning how to play games like alpha-go showing better-than-human performance. Deep Q-learning networks (DQN) and Deep Deterministic Policy Gradient (DDPG) are two such methods that have shown state-of-the-art results in recent times. Among the many variants of RL, an important class of problems is where the state and action spaces are continuous -autonomous robots, autonomous vehicles, optimal control are all examples of such problems that can lend themselves naturally to reinforcement based algorithms, and have continuous state and action spaces. In this paper, we adapt and combine approaches such as DQN and DDPG in novel ways to outperform the earlier results for continuous state and action space problems. We believe these results are a valuable addition to the fast-growing body of results on Reinforcement Learning, more so for continuous state and action space problems.
This paper combines elements of two existing reinforcement learning approaches, namely, Deep Q-learning Networks (DQN) with Prioritised Experience Replay (PER) and Deep Deterministic Policy Gradient (DDPG) to propose the Prioritized Deep Deterministic Policy Gradient (PDDPG) algorithm. The problem is interesting and there is a nice review of relevant work. The algorithm has a limited novelty with a simple modification of the DDPG algorithm to add the PER component. Experiment results show improvements in certain simulation environments. However, the paper lacks insight on how and why results are improved on some settings while performing worse than the others. Detailed comments are as follows:
1. Algorithm 1 is not self-contained. Yes, I understand that it is a slight modification to DDPG with changes being Line 11 and 16. But p_i^alpha is not defined anywhere in Algorithm 1. How the transition probabilties are updated on Line 16 is also not clear to me.
2. It would be better if multiple simulation runs on the same experiment can be performed to have a more reliable display of performance.
3. Section 6 is on Parameter Space Noise for Exploration. This is not the authors' proposed work so it is strange to have a separate section here. In the end of Section 1, the authors wrote that "We then use the concept of parameter space noise for exploration and show that this further improves the rewards achieved." This seems to be a bold claim from the varying performance displayed in Figure 2-4. Similar to Comment 2, more simulation runs and statistical tests need to be conducted to support this claim.