Deep reinforcement learning algorithms have proven successful in a variety of domains. However, tasks with sparse rewards remain challenging when the state space is large. Goal-oriented tasks are among the most typical problems in this domain, where a reward can only be received when the final goal is accomplished. In this work, we propose a potential solution to such problems with the introduction of an experience-based tendency reward mechanism, which provides the agent with additional hints based on a discriminative learning on past experiences during an automated reverse curriculum. This mechanism not only provides dense additional learning signals on what states lead to success, but also allows the agent to retain only this tendency reward instead of the whole histories of experience during multi-phase curriculum learning. We extensively study the advantages of our method on the standard sparse reward domains like Maze and Super Mario Bros and show that our method performs more efficiently and robustly than prior approaches in tasks with long time horizons and large state space. In addition, we demonstrate that using an optional keyframe scheme with very small quantity of key states, our approach can solve difficult robot manipulation challenges directly from perception and sparse rewards.
This paper proposes a new method for reverse curriculum generation by gradually reseting the environment in phases and classifying states that tend to lead to success. It additionally proposes a mechanism for learning from human-provided "key states".
The ideas in this paper are quite nice, but the paper has significant issues with regard to clarity and applicability to real-world problems:
First, it is unclear is the proposed method requires access only high-dimensional observations (e.g. images) during training or if it additionally requires low-dimensional states (e.g. sufficient information to reset the environment). In most compelling problems settings where a low-dimensional representation that sufficiently explains the current state of the world is available during training, then it is also likely that one can write down a nicely shaped reward function using that state information during training, in which case, it makes sense to use such a reward function. This paper seems to require access to low-dimensional states, and specifically considers the sparse-reward setting, which seems contrived.
Second, the paper states that the assumption "when resetting, the agent can be reset to any state" can be satisfied in problems such as real-world robotic manipulation. This is not correct. If the robot could autonomously reset to any state, then we would have largely solved robotic manipulation. Further, it is not always realistic to assume access to low-dimensional state information during training on a real robotic system (e.g. knowing the poses of all of the objects in the world).
Third, the experiments section lacks crucial information needed to understand the experiments. What is the state, observation, and action space for each problem setting? What is the reward function for each problem setting? What reinforcement learning algorithm is used in combination with the curriculum and tendency rewards? Are the states and actions continuous or discrete? Without this information, it is difficult to judge the merit of the experimental setting.
Fourth, the proposed method seems to lack motivation, making the proposed scheme seem a bit ad hoc. Could each of the components be motivated further through more discussion and/or ablative studies?
Finally, the main text of the paper is substantially longer than the recommended page limit. It should be shortened by making the writing more concise.
Beyond my feedback on clarity and significance, here are further pieces of feedback with regard to the technical content, experiments, and related work:
I'm wondering -- can the reward shaping in Equation 2 be made to satisfy the property of not affecting the final policy? (see Ng et al. '09) If so, such a reward shaping would make the method even more appealing.
How do the experiments in section 5.4 compare to prior methods and ablations? Without such a comparison, it is impossible to judge the performance of the proposed method and the level of difficulty of these tasks. At the very least, the paper should compare the performance of the proposed method to the performance a random policy.
The paper is missing some highly relevant references. First, how does the proposed method compare to hindsight experience replay? [1] Second, learning from keyframes (rather than demonstrations) has been explored in the past [1]. It would be preferable to use the standard terminology of "keyframe".
[1] Andrychowicz et al. Hindsight Experience Replay. 2017
[2] Akgun et al. Keyframe-based Learning from Demonstration. 2012
In summary, I think this paper has a number of promising ideas and experimental results, but given the significant issues in clarity and significance to real world problems, I don't think that the current version of this paper is suitable for publication in ICLR.
More minor feedback on clarity and correctness:
- Abstract: "Deep RL algorithms have proven successful in a vast variety of domains" -- This is an overstatement.
- The introduction should be more clear with regard to the assumptions. In particular, it would be helpful to see discussion of requiring human-provided keyframes. As is, it is unclear what is meant by "checkpoint scheme", which is not commonly used terminology.
- "This kind of spare reward, goal-oriented tasks are considered the most difficult challenges" -- This is also an overstatement. Long-horizon tasks and high-dimensional observations are also very difficult. Also, the sentence is not grammatically correct.
- "That is, environment" -> "That is, the environment"
- In the last paragraph of the intro, it would be helpful to more clearly state what the experiments can accomplish. Can they handle raw pixel inputs?
- "diverse domains" -> "diverse simulated domains"
- "a robotic grasping task" -> "a simulated robotic grasping task"
- There are a number of issues and errors in citations, e.g. missing the year, including the first name, incorrect reference
- Assumption 1: \mathcal{P} has not yet been defined.
- The last two paragraphs of section 3.2 are very difficult to understand without reading the method yet
- "conventional RL solver tend" -> "conventional RL tend", also should mention sparse reward in this sentence.
- Algorithm 1 and Figure 1 are not referenced in the text anywhere, and should be
- The text in Figure 1 and Figure 3 is extremely small
- The text in Figure 3 is extremely small