DOM-Q-NET: GROUNDED RL ON STRUCTURED LANGUAGE
Building agents to interact with the web would allow for significant improvements in knowledge understanding and representation learning. However, web navigation tasks are difficult for current deep reinforcement learning (RL) models due to the large discrete action space and the varying number of actions between the states. In this work, we introduce DOM-Q-NET, a novel architecture for RL-based web navigation to address both of these problems. It parametrizes Q functions with separate networks for different action categories: clicking a DOM element and typing a string input. Our model utilizes a graph neural network to represent the tree-structured HTML of a standard web page. We demonstrate the capabilities of our model on the MiniWoB environment where we can match or outperform existing work without the use of expert demonstrations. Furthermore, we show 2x improvements in sample efficiency when training in the multi-task setting, allowing our model to transfer learned behaviours across tasks.
Caveat: I am an emergency reviewer filling in for someone that fell through on their commitment to review for ICLR.  The framing of this paper is quite outside my typical area, so I am not super familiar with the related work here, nor do I have time to get familiar with it for this last-minute review.                                                                                                                                                                             
                                                                                                     
This paper presents a new model for deep reinforcement learning on web pages, where the system is given a goal (stated in text) and is supposed to interact with the web page (through clicking and entering text) in order to achieve that goal.  The supervision is a positive reward when the sequence of actions taken matches the goal.  The novel model presented in this paper is a modular Q function that incorporates graph embeddings of the web page's DOM, as well as similarity scores between elements in the DOM with words in the goal.
                                                                                                     
Just judging the presentation of the paper, it looks sound.  The methods seem reasonable (very similar to methods that are known to work well on related problems; more on that below), and the experiments look to be well done.  The paper is reasonably well written.  I don't know the RL community well enough to know how impactful this particular piece of work would be there - it's a new model architecture, basically, that gives improved performance.  I'd probably give a similar paper in my area a 3.5-4 out of 5 for an ACL conference.  The one major drawback I see in this paper is that it is _so_ similar to work on semantic parsing, but doesn't realize it.
                                                                                                     
I am not a "reinforcement learning" researcher, though I am a "semantic parsing" researcher.  The problem statement in this paper reads to me exactly like a semantic parsing problem: map a piece of text to a statement in some formal language.  In this case, the "statement" is a sequence of actions on the DOM of a web page.  The web page is possibly unseen at test time (the particulars of the data setup weren't totally clear to me), so the model has to be able to handle linking words in the sentence to pieces of the DOM in a way that doesn't rely on having seen those DOM elements during training.  This setup seems almost identical to the WikiTableQuestions dataset (Pasupat and Liang 2015), which has seen several RL-inspired works recently (e.g., https://arxiv.org/abs/1807.02322).  The way that the authors propose to use attention scores in the "global module" is _very_ similar to the linking mechanism proposed by Krishnamurthy, Dasigi and Gardner (EMNLP 2017) for WikiTableQuestions, and the way that the "word-token selection" only allows words in the goal sentence is very reminiscent of Chen Liang's language for parsing questions in WikiTableQuestions, which has similar restrictions for similar reasons.
                                                                                                     
I think the main difference between what we call "weakly-supervised semantic parsing" and what you call "deep reinforcement learning" is that semantic parsing leverages the fact that we know the language we're parsing into, so we don't need to use model-free RL methods like Q-learning.  We know the model, so we can be much smarter about learning.  Again, I'm not super familiar with the tasks you're looking at here, but I'm pretty sure there are much better _supervised_ learning techniques that you could apply to these problems.
                                                                                                     
All of this is to say that the methods proposed here look _very_ similar to methods that have been studied for quite a while in the semantic parsing literature (I gave only recent references above, but the basic problems go back decades; e.g., http://aclweb.org/anthology/P09-1010, or http://www.cs.utexas.edu/~ml/papers/senior-aaai-2008.pdf).  Yet this paper only cites recent deep RL papers.  I think the authors would benefit greatly from familiarizing themselves with this literature.  I think the semantic parsing community would also benefit from this, as there are surely ideas in the deep RL community that we could benefit from, too.  But the two communities don't really talk to each other much, it seems, even though in some cases we are working on _very_ similar problems.
                                                                                                     
So, to summarize: the paper seems reasonable enough.  I'm guessing that the RL community would find it at least moderately interesting, and it appears well written and well executed.  My one concern is that it's totally oblivious to the fact that it's sitting right next to a well-established literature that could probably teach it a thing or two about mapping language to actions.
--------------
After seeing the authors engage at least a little with the related semantic parsing literature, I've increased my score to a 7.