Obtaining policies that can generalise to new environments in reinforcement learning is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new environments. We propose a grounded policy learning problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason over a language goal, relevant dynamics described in a document, and environment observations. We procedurally generate environment dynamics and corresponding language descriptions of the dynamics, such that agents must read to understand new environment dynamics instead of memorising any particular information. In addition, we propose txt2π, a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2π generalises to new environments with dynamics not seen during training via reading. Furthermore, our model outperforms baselines such as FiLM and language-conditioned CNNs on RTFM. Through curriculum learning, txt2π produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.
First of all, I should acknowledge that this is a last-minute emergency review. Second, I should also acknowledge that I haven’t actively followed the specific research topic handled in this paper, although I work quite a lot in reinforcement learning and dialogue systems. The last paper I read on the topic was Branavan et al’s “Learning to win by reading manuals in monte-carlo framework” (ACL 2011).
This paper is about language-conditioned reinforcement learning, where the agent is required to perform machine reading of documents to learn policies to solve a task in new environments. The contribution of this paper is seemingly two fold: first, a new benchmark test suite is proposed (named “read to fight monsters” funny acronym), and second, an improved neural model (named “txt2pi”) which extends FiLM proposed for related tasks. Various experiments on txt2pi are conducted to show how the model performs against baselines, as well as it behaves under different learning settings.
Overall, this paper makes a good contribution to the research on deep models for language-conditioned reinforcement learning, but I see a number of things I miss in this paper. 
The first one is about the test suite RTFM. The authors claim that it poses a new challenge that the agent has to understand the language specification of the goal, the environment dynamics, and the observations *all together* (compared to prior work that misses one or two), I am not confident that it is not challenging enough. 
- The main concern I have is that they are all machine-generated, and soon or later, this kind of test suite could be beaten-to-death. A more ideal test suite would be the texts collected by humans, aiming for more diversity, ambiguity, and sometimes incorrectness. 
- The RTFM would combinatorily generate a large number of environments, but still, they all look pretty simple. It would be interesting to have “partially specified” documents, such as “lightning monsters are known to be weak against grandmaster’s or soldier’s weapons, but it is not specifically known”. It would be interesting to have partially observable environments, where the agent has to do some information gathering in the environment to behave optimally.
The second one is about txt2pi model proposed in the paper.
- txt2pi adequately extends FiLM to handle multiple sources of information, but what is the lesson being learned? 
I find the discussions in page 15 very enlightening. Could you move the content to the main text and defer the curriculum learning details to the appendix?
In summary, this paper proposes an interesting test suite for language-conditioned reinforcement learning, but I wish it is more realistic in the language data and more complex in the RL tasks. 
--- 
I have updated my rating based on the reviewer response. Thank you for clarifying some of my concerns. I am still a bit reserved because of the use of synthetic corpus. However, at the same time, I think there could be a lot of follow-up papers based on this work.