FROM LANGUAGE TO GOALS: INVERSE REINFORCE- MENT LEARNING FOR VISION-BASED INSTRUCTION FOLLOWING
Reinforcement learning is a promising framework for solving control problems, but its use in practical situations is hampered by the fact that reward functions are often difficult to engineer. Specifying goals and tasks for autonomous machines, such as robots, is a significant challenge: conventionally, reward functions and goal states have been used to communicate objectives. But people can communicate objectives to each other simply by describing or demonstrating them. How can we build learning algorithms that will allow us to tell machines what we want them to do? In this work, we investigate the problem of grounding language commands as reward functions using inverse reinforcement learning, and argue that language-conditioned rewards are more transferable than language-conditioned policies to new environments. We propose language-conditioned reward learning (LC-RL), which grounds language commands as a reward function represented by a deep neural network. We demonstrate that our model learns rewards that transfer to novel tasks and environments on realistic, high-dimensional visual environments with natural language commands, whereas directly learning a languageconditioned policy leads to poor performance.
Summary:
This paper proposes learning reward functions via inverse reinforcement learning (IRL) for vision-based instruction following tasks like "go to the cup". The agent receives the language instruction (generated via grammar templates) and a set of four images (corresponding to four cardinal directions) from virtual cameras mounted on the agent as input at every time step and its aim is either to 1. navigate to the goal location (navigation task) or 2. move an object from one place to another (pick task). 
The really interesting part in this paper is learning reward functions such that they generalize across different tasks and environments (e.g. indoor home layouts). This differentiates it from the standard IRL setting where reward functions are learnt and then policies optimized on this reward function on the *same* environment. 
In order to generalize across tasks and environments a slight modification to the max-ent IRL gradient equations are made: 1. Similar to meta-learning the gradient is taken with respect to multiple tasks (in a sampling-based manner) and 2. Making the reward function a function of not just states and actions but also language context. The overall algorithm (Algorithm 1) is simple and the critical step of computing an optimal policy to compute the IRL gradient is done by assuming that one has access to full state and dynamics and essentially running a planner on the MDP. This assumption is not unreasonable since in a simulator one has access to the full dynamics and can hence one can compute the optimal trajectories by planning. 
Experiments are presented on the SUNCG dataset of indoor environments. Two baselines are presented: One using behavior cloning (BC) and an oracle baseline which simply regresses to the ground truth reward function which is expected to be an upper bound of performance. Then DQN is used (with and without reward shaping) using the learnt reward functions to learn policies which are shown to have better performance on different tasks. 
Comments and Questions:
- The paper is generally well-written and easy to understand. Thanks!
- The idea of using IRL to learn generalizable reward functions to learn policies so as to aid transfer between environments in such vision-language navigation tasks is interesting and clearly shows benefits to behavior cloning.
- One of my main concerns (and an interesting question that this paper naturally raises) is how does this approach compare to imitation learning (not vanilla behavior cloning which is straight-up supervised learning and has been theoretically and empirically shown to have worse performance due to distribution shifts. See Ross and Bagnell, 2011, Ross, Gordon, Bagnell 2012 (DAgger, Ross and Bagnell 2014 (AggreVate), Chang et al., 2015 (LOLS), etc). If the same budget of 10 demonstrations per environment is used via DAgger (where say each iteration of DAgger gets say 2 or 3 demonstrations until the budget is exhausted) how does it compare? Note online version of DAgger has already been used in similar settings in "Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments" by Anderson et al, CVPR 2018. The main difference from their setting is that this paper considers higher level tasks instead of taking as input low-level turn-by-turn language inputs. 
- The following papers are relevant and should be cited and discussed:
"Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environments" by Anderson et al, CVPR 2018.
"Embodied Question Answering", Das et al, CVPR 2018.
Update:
------------
After looking at other reviews and author rebuttals to all reviews I am raising my grade.