LEARNING TO UNDERSTAND GOAL SPECIFICATIONS BY MODELLING REWARD
Recent work has shown that deep reinforcement-learning agents can learn to follow language-like instructions from infrequent environment rewards. However, this places on environment designers the onus of designing language-conditional reward functions which may not be easily or tractably implemented as the complexity of the environment and the language scales. To overcome this limitation, we present a framework within which instruction-conditional RL agents are trained using rewards obtained not from the environment, but from reward models which are jointly trained from expert examples. As reward models improve, they learn to accurately reward agents for completing tasks for environment configurations-and for instructions-not present amongst the expert data. This framework effectively separates the representation of what instructions require from how they can be executed. In a simple grid world, it enables an agent to learn a range of commands requiring interaction with blocks and understanding of spatial relations and underspecified abstract arrangements. We further show the method allows our agent to adapt to changes in the environment without requiring new expert examples.
==========
Update
==========
Upon reviewing the paper revision and the author comments to my and the other reviewers' comments, I will revise my suggestion to that of acceptance. As I said in my summary, my primary concern was novelty with respect to prior work which the authors have clarified. They have also increased the rigor of their experimental results by providing variances in the plots.
I think this work will be of interest to the community.
==========
Strengths:
==========
- The problem of learning to predict state rewards given language in interesting and useful. 
- The proposed AGILE framework is intuitively simple and works with any existing RL framework.
- With the models and tasks explored in this paper, the approach does seem to learn to evaluate whether a state matches the instructions quite well. 
- The writing is very clear and direct. 
==========
Concerns:
==========
[A] The discussion of differences to the closely related GAIL methodology is left until the related work after experiments. Given the similarities between GAIL and AGILE, this seems too late. The authors list three major differences between AGILE and GAIL:
	
1) AGILE is conditioned on a goal specification, language in this case. GAIL is unconditioned and trained for one task.
2) AGILE takes only the final/goal state rather than a trajectory like in GAIL.
3) AGILE discretizes the discriminator probability when assigning reward, GAIL does not.
Some concerns about each:
		
1) This is an interesting and fair difference but also a necessary and somewhat obvious modification to GAIL in tasks with explicit goal-specification. 
2) This does not seem like an improvement, but rather a loss of generality. The authors justify this change saying "in AGILE the reward model observes only states s_i (either goal states from an expert, or states from the agent acting on the environment) rather than traces (s1, a1),(s2, a2), . . ., learning to reward the agent based on “what” needs to be done rather than according to “how” it must be done." 
In many real applications, the how is deeply important. For instance, navigation in the world is both a "what" (arrive at location X) and a "how" (in fastest time without hitting anything or in such a way that humans aren't frightened). Further, the trace includes the final state such that the "what" is recoverable in instances where the "how" is unimportant, as in the set of tasks presented in this paper. 
3) Letting the paper speak on this subject: "We considered this change of objective necessary because the GAIL-style reward would take arbitrarily low values for intermediate states visited by the agent, as the reward model will be confident as those are not goal states. The binary reward in AGILE carries a clear message to the policy that all non-goal states are equally undesirable." Firstly, all non-goal states are not equally undesirable in that some lead more easily to goal states though it is fair to argue this should be learned by the policy through expected reward. My primary gripe is the footnote following these sentences which says: "We tried values other than 0.5 for the binarization threshold, as well as not binarizing and using Dφ(c, st) directly as the reward. We got similar but slightly worse results." This seems to imply that this difference does not matter significantly, especially if different thresholds received significantly different hyperparameter tuning effort or were not conducted under multiple runs of random seeds.
A pessimistic summary would place AGILE to be a conditional GAIL with reduced ability to represent intermediate or trajectory based rewards and a possibly slightly helpful reward discretization scheme. Don't get me wrong, I think an conditional extension to GAIL is interesting and worth sharing with the community. However, this discussion comes very late and includes a design decisions (2/3) that I find poorly justified in text and completely unjustified experimentally. 
I would like to hear from the authors if any of these criticisms are inaccurate. I would also welcome experiments evaluating the effect of these design decisions.
[B] In 3.2 its reported that each experiment was repeated five times however the presented results are not described as means and no variances are shown. I would like to see the results plots with shaded variances from at least 5 runs with differing random seeds. 
[C] Unless I'm mistaken, the proposed architecture could also be trained with reward prediction. It would be interested in that case to see if improvement seen between A3C and A3C-AGILE extend to A3C-RP and A3C-RP-AGILE. As the authors note, the AGILE framework simply changes the source of the reward and is amicable to any RL approach. I would like to see this comparison.
[D] The reward generalization experiments seemed surprising to me. The policy was fine-tuned on the test environments but only improved from 52% to 69.3%. Trying to think about this more, I'm having trouble disentangling whether this implies poor generalization of the reward function or increased difficulty in policy learning. Could the authors provide the A3C and A3C-RP baselines for this experiment to help clarify?
[E] Just a Curiosity: What exactly is done in L2 weight clipping? (Training details in supplement)
[F] Just a Thought: In the reward-prediction (RP) setting, both the RP model and the policy share parameters. It would be possible with such an architecture to still apply the AGILE loss and I would be curious to see if this leads to interesting changes in performance. I understand that one of the advantages to learning a separate reward model is to generalize to new policies, but it is unclear if this approach would generalize less well (and finding it out would be cool!)
==========
Overview:
==========
I think extending generative adversarial imitation learning to a task-conditional setting a cool step made even more interesting in this work by having the task-specification be in compositional language. Further, the results and analysis are generally interesting though I do note some weaknesses above. Aside from some questions about the experiments, I'm mostly concerned about the positioning of the paper -- specifically with respect to prior work.  I'm looking forward to hearing from the authors and other reviewers.