Developing agents that can learn to follow natural language instructions has been an emerging research direction. While being accessible and flexible, natural language instructions can sometimes be ambiguous even to humans. To address this, we propose to utilize programs, structured in a formal language, as a precise and expressive way to specify tasks. We then devise a modular framework that learns to perform a task specified by a program -as different circumstances give rise to diverse ways to accomplish the task, our framework can perceive which circumstance it is currently under, and instruct a multitask policy accordingly to fulfill each subtask of the overall task. Experimental results on a 2D Minecraft environment not only demonstrate that the proposed framework learns to reliably accomplish program instructions and achieves zero-shot generalization to more complex instructions but also verify the efficiency of the proposed modulation mechanism for learning the multitask policy. We also conduct an analysis comparing various models which learn from programs and natural language instructions in an end-to-end fashion.
This paper presents a reinforcement learning agent that learns to execute tasks specified in a form of programs with an architecture consisting of three modules. The (fixed) interpreter module interprets the program, by issuing queries to a (pre-trained) vision module and giving goals to a policy module that executes them in the environment. The paper also introduces a policy modulation technique, with the goal of modulating the current state with the expected (symbolic) goal. The model is evaluated on a 2D approximation of Minecraft, where it outperforms a set of baselines. In addition, the authors modified the program dataset and re-expressed it in terms of natural language and showed that the baselines perform better with programs than with the natural language instructions.
Though I think the general idea of the paper is worth exploring, I find the concrete contribution of the paper a bit thin to the point that I am hesitant to recommend this paper for acceptance. Allow me to explain my objections:
First and foremost, this work is very close to work by Denil et al where the execute (in a differentiable fashion) programs in an RL agent. Their work does not have a discrete interpreter per-se, but it does have a differentiable execution of commands. The major difference between these two works would be that Denil et al do not have the vision module (they do mention learning from pixels as future work).
However, that is not entirely true. The model presented here uses a pretrained vision module, which by itself is not a problem and is used in related work [1], but this vision module does not operate on visual input but the symbolic representation of the map. A crucial thing here is that if all of a sudden we want to include a new object on the map, the model won’t be able to use some learned similarity since it would require introducing a new object (slice of the input), as it would should it have been trained on pixels. So technically speaking, this is not a vision module but a symbolic state processing module.
Then, the modulation mentioned in the paper does not seem particularly novel. Sure, the exact architecture of the model is probably unique, but the idea of modulating a state with a goal is not and has been seen in other work such as [2] and [3] among others. The paper does not mention why, for example, this modulation technique is useful and why any other similar architecture would not be as successful, nor does it mention related modulation techniques in other work.
A big issue I have with the evaluation in the paper is that I do not see the benefit of having the experiments with natural language at all. The focal point of the paper are agents able to execute tasks in the form of programs. The (though manually) generated natural language instructions from those same programs cannot even be used by the proposed agent as there is no natural language interpreter, so they are a dangling part of the paper which is there just to showcase that programs should be easier for baselines to learn to execute than natural language (the hypothesis would be that a simpler and more structured/formal language is easier to learn than the natural language). Hence the seq-LSTM results on language which are a tad lower than the results on programs are expected, though the performance of transformers is the opposite---they are better on language, and that is something that is puzzling and left unexplained, as well as the unexpectedly low performance of Tree-RNNs. One would expect them to perform a bit better than LSTMs, but that might be contingent on the size of the dataset more than the structure of the inputs. However, none of these curious findings have been explained.
Moreover, the comparison to baselines is not particularly fair as these baselines had to learn the symbolic state interpretation, whereas your model did not. You could have provided the same to the baselines for a better comparison.
In addition to that, 5.4.2 goes into detail of analysing the baselines, and ignoring the proposed model. Why didn’t you include the same statistics for your agent in Figure 5 and said subsubsection?
The paper is missing some notable related work:
- S.R.K. Branavan’s work (all but one cited language-instructed agent papers are post 2017) as well as [4]
- object-oriented and hierarchical RL
- [5], where they train the neural programmer interpreter from the final reward only, which brings NPI close to this work
Questions:
- Figure 5 - There’s a mark for Tree-RNN, but Tree-RNNs are not in any figure. Why didn’t you plot the performance of your model?
- The setup naturally induces a curriculum - how does it do that if the programs are randomly sampled?
- You say that your model learns to comprehending the program flow. I’m not sure I would agree with that because what your model learns is to execute single commands. From what it seems, the interpreter is the only part of the model (which is fixed btw) which sees the control flow, whereas the policy just executes singular commands. Did you mean something else by that statement?
Minor issues:
- You say twice interpreter (i.e. compiler). Given that they’re not the same, and you’re using an interpreter, I suggest omitting the word compiler.
- Figure 2 is lacking detail. There is no difference between r and i (both being positive integers) other than their descriptions, - operators agent[_], env and is_there lack parameters (non-terminal nodes), and where’s river, bridge, etc?
[1] COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration
[2] FeUdal Networks for Hierarchical Reinforcement Learning
[3] Universal value function approximators
[4] Vogel et al Learning to follow navigational directions
[5] Improving the Universality and Learnability of Neural Programmer-Interpreters with Combinator Abstraction