LEARNING TO NAVIGATE THE WEB
Learning in environments with large state and action spaces, and sparse rewards, can hinder a Reinforcement Learning (RL) agent's learning through trial-anderror. For instance, following natural language instructions on the Web (such as booking a flight ticket) leads to RL settings where input vocabulary and number of actionable elements on a page can grow very large. Even though recent approaches improve the success rate on relatively simple environments with the help of human demonstrations to guide the exploration, they still fail in environments where the set of possible instructions can reach millions. We approach the aforementioned problems from a different perspective and propose guided RL approaches that can generate unbounded amount of experience for an agent to learn from. Instead of learning from a complicated instruction with a large vocabulary, we decompose it into multiple sub-instructions and schedule a curriculum in which an agent is tasked with a gradually increasing subset of these relatively easier sub-instructions. In addition, when the expert demonstrations are not available, we propose a novel meta-learning framework that generates new instruction following tasks and trains the agent more effectively. We train DQN, deep reinforcement learning agent, with Q-value function approximated with a novel QWeb neural network architecture on these smaller, synthetic instructions. We evaluate the ability of our agent to generalize to new instructions on World of Bits benchmark, on forms with up to 100 elements, supporting 14 million possible instructions. The QWeb agent outperforms the baseline without using any human demonstration achieving 100% success rate on several difficult environments.
UPDATE:
Thank you to the authors for a comprehensive response.  I have increased my score based on these changes.  I apologize for the misunderstanding about ArXiV papers and indeed the authors are correct on that point.  Thank you as well for reporting the learning speeds.  As you mentioned, they confirm our intuitions and complete the picture of the algorithm’s behavior.  The addition of pseudo-code does make the paper and algorithm easier to follow.  Thank you for adding it.  The rewritten section 5 is indeed much easier to follow and makes the coordination between the agents clear.  Seeing that the instructor is a fixed policy resolves the game theoretic issue form the original review.
Summary:
The paper proposes a deep reinforcement learning approach to filling out web forms, called QWeb.  In addition to both deep and shallow embeddings of the states, the authors evaluate various methods for improving the learning system, including reward shaping, introducing subgoals, and even a meta-learning algorithm that is used as an instructor.  These variations are tested in several environments and basic QWeb is shown to outperform the baselines and many of the adaptations perform even better than that in more complex domains.
Review:
Overall, the problem the paper considers is important and their results seem significant.  The authors have derived a novel architecture and are the first to tackle the problem of filling in web forms at this scale with an autonomous learning agent rather than one that is taught mostly by demonstration.  
The related work section is very well written with topical references to recent results and solid differentiations to the new algorithm.  However, I see many references in the paper are not from peer reviewed conferences or journals.  Unless absolutely necessary, such papers should not be cited because they have not been properly peer reviewed.  If the papers cited have actually been in a conference or journal, please add the correct attribution.
The experiments seem well conducted.  I liked that each new addition to the algorithm was tested incrementally in Figure 7 to give a realistic view of the gains introduced by each change.  I also thought the earlier comparisons to the baselines were well done and I liked that they were done against modern cutting-edge LfD demonstrations.  The only thing I would have liked to seen beyond these results are actual learning curves showing, after X iterations, what percentage of the tasks could be completed.  I suspect that in many domains the baseline LfD techniques are learning much faster since learning from teachers tends to be more targeted and sample efficient.  Learning curves would show us whether or not this is the case. 
The weakest part of the paper was the description of the instructor network and the Meta-training in general.  This portion seemed ill-described and largely speculative, despite the promising results in Figure 7.  In particular, Section 5 is very unclear on how exactly the Meta-Learning works.  Pseudocode is definitely needed in this portion well beyond the quick descriptions in Figure 4 and 5, which I could not understand, despite multiple readings.  I suggest eliminating those figures and providing concrete pseudo—code describing the meta learning and also addressing the following open questions in the text:
•	Why is a rule based randomized policy good to learn from?  How is this different from learning from demonstration in the baselines?
•	How is a “fine grained signal” generated?  What does that mean?  Is it a reward?
•	In Section 5.1, are there two RL agents, an instructor and a learner with different reward functions?  If so, isn’t this becoming game theoretic and is this likely to converge in most scenarios?
•	What does Q_D^I actually represent?  Why is maximizing these values a good thing?
There are a few grammatical mistakes in the paper including:
Abstract – simpler environments -> simple environments
Abstract- with gradually increasing -> with a gradually increasing
Page 2 – generate unbounded -> generate an unbounded
Page 7 – correct value -> correct values
Page 9 – episode length -> episode lengths