Recent work has described neural-network-based agents that are trained to execute language-like commands in simulated worlds, as a step towards an intelligent agent or robot that can be instructed by human users. However, the instructions that such agents are trained to follow are typically generated from templates (by an environment simulator), and do not reflect the varied or ambiguous expressions used by real people. We address this issue by integrating language encoders that are pretrained on large text corpora into a situated, instruction-following agent. In a procedurally-randomized first-person 3D world, we first train agents to follow synthetic instructions requiring the identification, manipulation and relative positioning of visually-realistic object models. We then show how these abilities can transfer to a context where humans provide instructions in natural language, but only when agents are endowed with language encoding components that were pretrained on text-data. We explore techniques for integrating text-trained and environment-trained components into an agent, observing clear advantages for the fully-contextual phrase representations computed by the well-known BERT model, and additional gains by integrating a self-attention operation optimized to adapt BERT's representations for the agent's tasks and environment. These results bridge the gap between two successful strands of recent AI research: agent-centric behavior optimization and text-based representation learning.
The authors present a method of transferring template-based instruction following agents to natural language instructions by using language encoders trained on large text corpora. They explore different ways of combining text-based language encoders with visual representations and compare them. They find that contextual phrase-based representations learned by BERT significantly improve the performance on natural language instructions. 
Strengths:
- The paper is written well, it is easy to understand and follow.
- The task setup is good, authors collect natural language instruction data from humans.
- The paper presents several language encoding methods for the task and systematically evaluates them in a scientific manner.  
- The experimental results indicate that it is possible to transfer an agent trained on template-based instructions to natural language instructions using language models trained on large text corpora. It is not necessary to train the agent on natural language instructions. I find this result important and useful.
Weaknesses:
- The paper lacks significant technical novelty. It essentially combines known reinforcement learning based instruction following agents with known language models. The different ways of combining language encoding with visual representations are either trivial or adapted from prior work.
- A major concern is that the natural language instructions considered in the paper do not have much diversity with respect to language. The paper only considers lifting and putting tasks and trains a separate model for both the tasks.  
-- The lifting task always uses the verb 'lift' and replaces the object word with synonyms or referring expressions. There are 80 objects in the lifting task and I suspect there are very few referring expressions for these objects and they mostly involve a synonym. Furthermore, at test time, the agent only needs to distinguish between 2 objects. The performance with random embedding is around 50% for this task and the best model is around 76% which means the agent is not recognizing the correct object around 50% of the time. 
-- For the putting task, authors consider synonyms for object words and natural instructions which involve changing the verb ‘put’. It seems like humans mostly use only 4 verb words for this task, ‘put’, ‘keep’, ‘move’, ‘place’. This might be an artifact of the examples given to the human annotators. In any case, this word is inconsequential as the agent always lifts one of 3 available objects on one of 2 fixed objects. 
- It seems like the most diversity is coming from synonyms which can probably be handled with a dictionary or wordnet rather than requiring a language model. There is also some prior work on handling synonyms (https://arxiv.org/pdf/1902.04546.pdf). I would have liked to see many more tasks and a multi-task learning model which is also able to distinguish between the task based on natural language instructions in addition to understanding object word synonyms and referring expressions. More objects would also help.
- The authors claim to tackle “more behavioural and environmental challenges than previous work”. I do not agree with this claim. It is true that this paper handles object interaction and natural language instructions in a partially observable setting, however, previous work has tackled other challenges which this work does not tackle. For example, Oh et al. 2017 generalize to new sequence of instructions, Hermann et al. 2017 and Chaplot et al. 2018 also handle compositionality and generalize to unseen instructions referring to new objects, Hermann et al. 2017 handle negation, Chaplot et al. 2018 handle instructions involving ‘largest’ or ‘smallest’ objects, Misra et al. 2018 handle more diverse natural language and so on.
- I wouldn’t call moving objects using high-level symbolic actions as ‘manipulation’. This is a whole research area in robotics involving taking low-level actions to move an object. Also, the environment used in the paper is not visually realistic in my opinion. It looks game-like and visual encoders trained in this environment are unlikely to generalize to the real world. This is fine as it is mostly irrelevant to handling natural language instructions, but authors should not claim visual realism and object manipulation in my opinion. 
Comments/Questions
- I do not understand the meaning and purpose of some actions. Why are there GRAB + actions? Doesn’t the object move with the agent once it is grabbed? What is SPIN_OBJECT? Why is it needed? It seems like there is no ‘place’ action, how does the agent place the object? I am guessing when the agent stops taking GRAB+ actions. If that is the case, then wouldn’t it be easier to just have Grab and Place actions rather than 16 GRAB+ actions?
- The meaning of ‘(sub)-’ in (sub)-word is not described.
- What is the probability of typo noise in the experiments?
- Many experimental details are missing. How long was the model trained for both the tasks? How many training samples/episodes? What were the hyperparameters used for reinforcement learning? Learning rate, optimizer, discount value and so on.
Updates after author response:
I have examined the author response and additional experiments carefully. I am maintaining my score due to the following reasons:
- The authors seem to agree that the paper does not provide any 'substantial algorithmic advance'.
-  The authors argue that the number of objects in the paper is much more than prior work. However, the focus of prior work (referenced by authors) was not to tackle natural language. Since the focus of this paper is to tackle natural language, I believe the number and diversity of objects and tasks need to be much higher than 80 objects and 2 tasks used in the paper.
- "We would language referring to an even wider range of motor-behaviours, e.g. more 'verbs'" -> I do not understand what the authors are trying to say here, but if "learning the motor programmes for such concepts in an environment" is not "the focus of the present paper", I believe the focus is only handling synonyms and referring expressions for objects. In my opinion, these are relatively easier to tackle, (for example using a dictionary or wordnet) than grounding 'verbs' into sequence of actions, which limits the scope of this paper further.
- The authors still claim to tackle object manipulation and visual realism which I do not agree with. I do not believe taking high-level "GRAB" action can be called object manipulation. The objects are taken from shape net with relatively realistic shapes, but neither the appearance of objects (textures, shadows) nor the relative arrangement of objects is realistic. In my opinion, a model trained in this environment has no hope of generalizing to the real-world. 
- I do not agree with the authors' argument against chance performance in the lifting task. From my experience, I believe an RL agent trained without any language input would perform at 50% if it receives a reward for lifting one of the two available objects. I do not understand why the authors chose to put only 2 objects in the environment. Why not put 5 or more objects?
- I appreciate authors' efforts towards multi-task learning results, however, tackling only 2 tasks is not convincing enough.