Knowledge-grounded dialogue is a task of generating an informative response based on both discourse context and external knowledge. As we focus on better modeling the knowledge selection in the multi-turn knowledge-grounded dialogue, we propose a sequential latent variable model as the first approach to this matter. The model named sequential knowledge transformer (SKT) can keep track of the prior and posterior distribution over knowledge; as a result, it can not only reduce the ambiguity caused from the diversity in knowledge selection of conversation but also better leverage the response information for proper choice of knowledge. Our experimental results show that the proposed model improves the knowledge selection accuracy and subsequently the performance of utterance generation. We achieve the new state-of-the-art performance on Wizard of Wikipedia (Dinan et al., 2019) as one of the most large-scale and challenging benchmarks. We further validate the effectiveness of our model over existing conversation methods in another knowledge-based dialogue Holl-E dataset (Moghe  et al., 2018).
Post author response edit: The authors did a good job of addressing many of the concerns of reviewers. I believe with these new results (esp to reviewer 4), they will have a stronger version for the camera ready. I'm bumping up my recommendation for this reason.
The authors propose a novel architecture for selecting knowledge in knowledge-grounded multi-turn dialogue. Their knowledge selection module uses a sequential latent variable scheme, and is claimed to be able to both handle diversity of knowledge selection in conversation as well as leverage the information from the response. The proposed model yields state of the art on two relevant benchmark datasets in terms of perplexity and F1, and scores higher in human evaluations as well.
The paper is relatively well-written, and the authors offer extensive insight into their approach, providing relevant equations and diagrams where necessary. The approach is well-motivated, and the experiments indicate that the model indeed helps on all evaluation fronts. A variety of baselines are considered and are shown to be inferior, in nearly every metric. I did not spend a lot of effort to try to understand their factorization, but the intuition makes sense, and their use of gumbel softmax provides a clear avenue to fix some of the hard-backprop issues apparent in the original Dinan et al. paper. I also appreciate the addition of the knowledge loss to the PostKS baseline: it’s a good effort to make the baseline as good as possible.
A few things bother me with the paper. The primary one is it concerns me a bit that the BERT pretraining does not improve significantly over the E2E transformer memnet (with just bert vocabulary). Unless I’m missing something, that model contained NO pretraining, so I would expect massive improvements. A sanity check there would be checking ppl with gold knowledge: if that doesn’t significantly improve, then I suspect the authors have something really weird about the pretraining or fine tuning. However, It also appears to me that replacing the GRU with a transformer in PostKS might be unfair: Transformers are way more data hungry than RNNs, and so both variants should be tried (though I would be okay with the loser being relegated to a footnote or appendix).
The human evaluations are not as convincing as the authors propose them to be, especially the difference in the “Test Seen” case. It is unclear to me why the authors believe that their “model’s merit would be more salient” in a multi-turn setting, and I think such an experiment would be good to show - or, at the very least, an indication that such an experiment was tried but results were not considered due to reasons X,Y, Z, etc. Overall, the rough improvement that is being provided in the first-stage of the two stage setting seems rather minor (23% -> 26% accuracy; 2.21 -> 2.35 human eval), and that the task remains extremely difficult
Questions
* I know the Dinan et al. models, at human evaluation time, hardcoded to not pick the same knowledge twice. Do you have a similar restriction? If not, maybe you can at least say that you manage to get rid of the need for that!
* As mentioned earlier, I would be curious to see multi-turn human evaluations. I understand this is expensive and a large ask.
* How are the examples in figure 3 chosen? Are they generally indicative of what is seen throughout the human evaluation?
* It would be useful to see a qualitative example of the model’s knowledge selection process when comparing to other models, rather than just the utterance generation (which is not the novel contribution of the paper).
Nits
* Small grammatical errors dealing with subject-verb agreement (plurals mostly).
* Using “-” instead of n/a in tables would make it mildly easier to see digest and see where metrics don’t make sense.