OF WIKIPEDIA: KNOWLEDGE-POWERED CONVERSATIONAL AGENTS
In open-domain dialogue intelligent agents should exhibit the use of knowledge, however there are few convincing demonstrations of this to date. The most popular sequence to sequence models typically "generate and hope" generic utterances that can be memorized in the weights of the model when mapping from input utterance(s) to output, rather than employing recalled knowledge as context. Use of knowledge has so far proved difficult, in part because of the lack of a supervised learning benchmark task which exhibits knowledgeable open dialogue with clear grounding. To that end we collect and release a large dataset with conversations directly grounded with knowledge retrieved from Wikipedia. We then design architectures capable of retrieving knowledge, reading and conditioning on it, and finally generating natural responses. Our best performing dialogue models are able to conduct knowledgeable discussions on open-domain topics as evaluated by automatic metrics and human evaluations, while our new benchmark allows for measuring further improvements in this important research direction.
This paper introduces a new dataset and method for chatbots. In contrast to previous work, this paper specifically probes how well a dialogue system can use external unstructured knowledge. 
Quality:
Overall, this is a very high-quality paper. The dataset is developed well, the experimental setup is well thought-through and the authors perform many ablation studies to test different model variants. The main criticism I have would be that the human evaluation is rather simple (rating 1-5), I would have expected more fine-grained categories, especially ones that relate to how much knowledge the system uses (I appreciate the "Wiki F1" metric, but that is an automatic metric). As it is, the human evaluation shows that most of their contributions are not appreciated by human annotators. Further, the paper ends a bit abruptly, I would have expected a more in-depth discussion of next steps.
Clarity:
The description of the work is clear in most places. I particularly like the abstract and introduction, which set up the rest of the paper nicely. In some places, perhaps due to space restrictions, method descriptions are a bit too short.
Originality:
The paper is fairly original, especially the aspect about specifically using external knowledge. The authors could have been more clear on how the work differs from other work on non-goal directed dialogue work though (last paragraph of related work section).
Significance:
The dataset is really well-developed, hence I believe many working in the dialogue systems community will re-use the developed benchmark and build on this paper.
More detailed comments:
- Missing reference for goal-oriented dialogue datasets: Wen et al. 2017, A Network-based End-to-End Trainable Task-oriented Dialogue System, https://arxiv.org/abs/1604.04562
- How does the proposed dataset differ from the Reddit and Wikipedia datasets discussed in the last paragraph of the related work section? This should be explained.
- Page 3, paragraph "Conversational Flow": what is the maximum number of turns, if the minimum is 5?
- Page 3, paragraph "Knowledge Retrieval": how were the top 7 articles and first 10 sentences choices made? This seems arbitrary. Also, why wasn't the whole text used?
- Page 3, paragraph "Knowledge Selection and Response Generation": how do you deal with co-reference problems if you only ever select one sentence at a time? The same goes for the "Knowledge Attention" model described in Section 4.
- Page 3, paragraph "Knowledge Selection and Response Generation": how often do annotators choose "no sentence selected"? It would be interesting to see more such statistics about the dataset
- Section 4.2: did you run experiments for BPE encoding? Would be good to see as this is a bit of a non-standard choice.
- Section 4.2: it would be good to explain the Cer et al. 2018 method directly in the paper
- Section 4.2: is there a reference for knowledge dropout? Also, it would be good to show ablation results for this.
- Section 5.1: why did you choose to pre-train on the Reddit data? There should be some more in-depth description of the Reddit dataset to motivate this choice.
- Section 5.1: what is the setup you use for multi-task learning on SQuAD? Is it just a hard parameter sharing model, or?
- Section 5.3: as stated above, the human evaluation is a little bit underwhelming, both in terms of setup and results. I'd expect a more fine-grained way of assessing conversations by humans, and also an explanation of why the retrieval performer without knowledge was assessed as being on par with the retrieval transformer memnet.
- Section 5.3: I assume higher=better for the human scores? This should be made explicit.
- Section 5.3: Have others used the "F1 overlap score"? If so, cite.
- Section 5.3: I don't understand the argument that the human evaluation shows that humans prefer more natural responses. How does it show that?
- Section 5.3: The Wiki F1 score is kind of interesting because it shows to what degree the model uses knowledge. But the side-by-side comparison with the human scores shows that humans don't necessarily prefer chatbot models that use a lot of knowledge. I'd expect this to be discussed, and suggestions for future work to be made accordingly.
- Section 6: The paper ends a bit abruptly. It's be nice to suggest future areas of improvement.