Meta reinforcement learning (meta-RL) is able to accelerate the acquisition of new tasks by learning from past experience. Current meta-RL methods usually learn to adapt to new tasks by directly optimizing the parameters of policies over primitive actions. However, for complex tasks which requires sophisticated control strategies, it would be quite inefficient to directly learn such a meta-policy. Moreover, this problem can become more severe and even fail in spare reward settings, which is quite common in practice. To this end, we propose a new meta-RL algorithm called meta goal-generation for hierarchical RL (MGHRL) by leveraging hierarchical actor-critic framework. Instead of directly generate policies over primitive actions for new tasks, MGHRL learns to generate high-level meta strategies over subgoals given past experience and leaves the rest of how to achieve subgoals as independent RL subtasks. Our empirical results on several challenging simulated robotics environments show that our method enables more efficient and effective meta-learning from past experience and outperforms state-of-the-art meta-RL and Hierarchical-RL methods in sparse reward settings.
This paper studies the problem of leveraging past experience to quickly solve new control tasks. The starting point (and perhaps the main contribution) is the observation that some tasks have similar high-level goals, while differing in how those goals are achieved. To that end, the paper introduces an meta-RL algorithm that, given a new task, attempts to solve it by adapting a high-level, goal-setting module, and learn a new, low-level policy to reach each commanded goal. The proposed method might be viewed as a combination of PEARL [Rakelly 19] and HAC [Levy 19]. The proposed method is compared against state-of-the-art hierarchical RL and meta-RL methods on four robotic manipulation tasks. The proposed method outperforms the baselines on each task.
While the proposed method is quite strong empirically, I am leaning towards rejecting this paper because many of the claims made in the paper are not empirically validated. While much emphasis is put on the hierarchical aspect of the algorithm, I don't think that the tasks used in the experiments require hierarchy to solve (see [Plappert 18]). In the introduction, the second claim is that the proposed method "focus[es] on meta learning the overall strategy … [and] provides a simpler and better way for meta RL." While the experiments show that the proposed method learns better than baselines, I don't think the paper show that the proposed method learns some sort of "overall strategy." I don't think that the proposed method is simpler than the baselines.
A second concern is that I'm confused about the experimental protocol. If the high-level policy outputs a desired XYZ position for the gripper (Section 5.1), how can the high-level policy indicate when the gripper should be closed to pick up the block? How is the reward function for the low-level policy defined? PEARL doesn't have access to this extra information (the reward function), right?
A third concern is the large number of grammatical errors in the paper.
I would consider increasing my review if (1) a new plot were added to visualize the commanded subgoals (I have a hunch that the high-level policy directly outputs the true goal, obviating the need for hierarchy and contradicting the claim that the method "learns to generate high-level meta-strategies over subgoals"); (2) the experimental protocol were clarified; and (3) the number of grammatical errors were significantly reduced.
Other comments:
* "inefficient to to directly learn such a meta policy" -- Why? Also, "to to" is repeated.
* "Deep Reinforcement learning" -- "Reinforcement" shouldn't be capitalized.
* "failing to generalize" -- Can you add a citation?
* "it would be quite inefficient to directly learn such a policy…": Doesn't [Plappert 18] do exactly this?
* "When the tasks distribution is much wider … these methods can hardly be effective…" -- Where is this claim substantiated? Also, "tasks" should be singular.
* "sparse reward settings which is" -> "sparse reward settings, which are"
* "the above mentioned problems" -> "the problems mentioned above"
* "our algorithm focus on meta learning … which provides a much simpler …" -- Where is it shown that the proposed method is simpler? Also, "focus" should be "focuses"
* "1991),which" -- Missing space
* "complex tasks which requires" -> "complex tasks that require"
* "Nachum et al … set of sub-policies" -- Run on sentence.
* "... human leverage…" -- Don't humans also transfer low-level knowledge across tasks, in addition to high-level knowledge? Also, "human" should be plural.
* "algorithms.The" -- Missing a space, I think.
* "PEARL leverages … latent variable Z" -- This sentence doesn't make sense as written.
* "z's" -- This should not be a possessive.
* "Good sample efficiency enables fast adaptation … and performs structured exploration …" -- Isn't the first part true by definition? Why does good sample efficiency perform structured exploration?
* "a goal is a 3-d vector" -- If the goal output by the high-level policy is the XYZ coordinates of the 
* "SAC, such non-hierarchical" -- Grammar doesn't make sense here.
* "such non-hierarchical RL method has been proved to perform badly before on …" -- Can you add a citation? Generally, "proved" is reserved for mathematical proofs.
* "In this paper, We have" -- "We" should not be capitalized.
------------------------ UPDATE AFTER AUTHOR RESPONSE ------------------
I thank the authors for at least reading the reviews. My concerns with experiments and clarify remain unaddressed, and are amplified by reading the other reviews. I therefore vote to "reject" this paper.