GUIDING POLICIES WITH LANGUAGE VIA META-LEARNING
Behavioral skills or policies for autonomous agents are conventionally learned from reward functions, via reinforcement learning, or from demonstrations, via imitation learning. However, both modes of task specification have their disadvantages: reward functions require manual engineering, while demonstrations require a human expert to be able to actually perform the task in order to generate the demonstration. Instruction following from natural language instructions provides an appealing alternative: in the same way that we can specify goals to other humans simply by speaking or writing, we would like to be able to specify tasks for our machines. However, a single instruction may be insufficient to fully communicate our intent or, even if it is, may be insufficient for an autonomous agent to actually understand how to perform the desired task. In this work, we propose an interactive formulation of the task specification problem, where iterative language corrections are provided to an autonomous agent, guiding it in acquiring the desired skill. Our proposed language-guided policy learning algorithm can integrate an instruction and a sequence of corrections to acquire new skills very quickly. In our experiments, we show that this method can enable a policy to follow instructions and corrections for simulated navigation and manipulation tasks, substantially outperforming direct, non-interactive instruction following.
UPDATE: I've increased my rating based on the authors' thorough responses and the updates they've made to the paper. However, I still have a concern over the static nature of the experimental environments.
=====================
This paper proposes the use of iterative, linguistic corrections to guide (ie, condition and adjust) an RL policy. A major challenge in learning language-guided policies is grounding the language in environment states and agent actions. The authors tackle this challenge with a meta-learning approach.
The approach is fairly complex, blending imitation and supervised learning. It operates on a training set from a distribution of virtual pick-move-place tasks. The policy to be learned operates on this set and collects data, via something close to DAgger, for later supervised learning on the task distribution. The supervised-learning data comprises trajectories augmented with linguistic subgoal annotations, which are referred to as policy "corrections." By ingesting its past trajectories and the correction information, the policy is meant to learn to solve the task and to ground the corrections at the same time, end-to-end. Correction annotations are derived from an expert policy.
The idea of guiding a policy through natural language and the requisite grounding of language in environment states and policy actions have been investigated previously: for example, by supervised pretraining on a language corpus, as in the cited work of Andreas et al. (2018). The alternative meta-learning approach proposed here is both well-motivated and original.
Generally, I found the paper clear and easy to read. The authors explain convincingly the utility of guiding policies through language, especially with respect to the standard mechanisms of reward functions (sparse, engineered) and demonstrations (expertise required). The paper is also persuasive on the utility of iterative, interactive correction versus a fully-specified language instruction given a priori. The meta-learning algorithm and training/test setup are both explained well, despite their complexity. On the other hand, most architectural details necessary to reproduce the work are missing, at least from the main text. This includes various tensor dimensions, the structure of the network for perceiving the state, etc.
I like the proposed experimental setting. It enables meta-learning on sequential decision making problems in a partially observable environment, which seems useful to the research community at large. Ultimately, however, this paper's significance is not evident to me, mainly because the proposed method lacks thorough experimental validation. No standard baselines are evaluated on the task (with or without meta-learning), nor is a detailed analysis of the learned policies undertaken. The ablation study is useful, and a good start, but insufficient in my opinion. Unfortunately, the results are merely suggestive rather than convincing.
Some things I'd like to see in an expanded results section before recommending this paper include:
- Comparison to an RL baseline that attempts to learn the full task, without meta-training or language corrections.
- Comparison to a baseline that learns from intermediate rewards. Instead of annotating data with corrections, you could provide +/- scalar rewards throughout each trajectory based on progress towards the goal (since you know the optimal policy). How effective might this be compared to using the corrections?
- Comparison to a baseline that does some kind of pretraining on the language corrections, as in Andreas et al. (2018).
- Quantification of how much meta-training data is required. What is the sample complexity like with/without language corrections?
I also have concerns about the need for near-optimal agents on each task -- this seems very expensive and inefficient. The expert policy is obtained via RL on each individual task using "ground truth" rewards. It is not specified what these rewards are, nor is it stated how near to optimal the resulting policy is nor how this nearness affects the overall meta-learning process.
Its unclear to me how the "full information" baseline processes and conditions on the full set of subgoals/corrections. Are they read as a single concatenated string converted to one vector by the bi-LSTM?
There also might be an issue with the experimental setup, unless I've misunderstood it. The authors state that "the agent only needs 2 corrections where the first correction is the location of the goal object and the second is the location of the goal square." But if the specific rooms, indicated by colors, do not change location from task to task (and they appear not to from all the figures), then the agent can learn the room locations during meta-training and these two "corrections" tell it everything it needs to know to solve the task.
Pros:
- Appealing, well-motivated idea for training policies via language.
- Clear, pleasant writing and good communication of a complicated algorithm.
- Good experimental setup that should be useful in other research (except for possible issue with static room locations).
Cons:
- The need for a near-optimal policy for each task. 
- Overall complexity of the training process.
- The so-called corrections are actually linguistic statements of subgoals computed from the optimal policy. There is much talk in the introduction of interactive policy correction by humans, which is an important goal and interesting problem, but the present paper does not actually investigate human interaction. This comes as a letdown after the loftiness of the introduction.
- Various details needed for reproduction are lacking. Maybe they're in the supplementary material; if so, please state that in the main text.
- Major lack of comparisons to alternative approaches.