Responding with knowledge has been recognized as an important capability for an intelligent conversational agent. Yet knowledge-grounded dialogues, as training data for learning such a response generation model, are difficult to obtain. Motivated by the challenge in practice, we consider knowledge-grounded dialogue generation under a natural assumption that only limited training examples are available. In such a low-resource setting, we devise a disentangled response decoder in order to isolate parameters that depend on knowledge-grounded dialogues from the entire generation model. By this means, the major part of the model can be learned from a large number of ungrounded dialogues and unstructured documents, while the remaining small parameters can be well fitted using the limited training examples. Evaluation results on two benchmarks indicate that with only 1/8 training data, our model can achieve the state-of-the-art performance and generalize well on out-of-domain knowledge.
This paper studies knowledge-grounded dialogue response generation in the low-resource setting. More precisely, it proposes a disentangled decoder consisting of three components: language model, context processor, and document reader. Disentangled decoder architecture provides a flexibility to train (or pre-train) different components on different data, making it convenient for low-resource setting. Overall, it is a sound idea for low-resource setting with generally positive experimental results, but limited in novelty in terms of the proposed architecture (similar to [1*, 2*]) and disentangling language and knowledge idea (similar to [3*]), lacks comparison with baselines for low-resource setting, and lacks discussion/reference of a few closely related works.
Here are some of my questions and concerns for the paper:
Great to have some ablations in terms of pre-training to see its effect on different components. However, it would be quite useful to also have ablations over components by completely dropping a component (like LM) while training decoding manager. It would also be useful to see some statistics/discussion on the effect of different components in the actual generation of responses. It might also be useful to include qualitative examples of the generated responses annotated with predictions of different components for each generated word.
In Figure-2 (c) and (d), some ablation results are reported for when pre-training is removed for each of the three components independently. It is interesting, though, to see that removing pre-training does not hurt (might even improve) the performance (esp. for Test Unseen) much for FULL training data case. Is there a particular reason for this observation? Also, it makes me curious how the proposed model would perform without pre-training any of the components. Would it already outperform the baselines discussed in the paper? If so, are these baselines strong enough (SOTA or close to SOTA) to help draw a meaningful conclusion from comparison with them? For example, how would fine-tuning a pre-trained MASS  [4*] perform and compare as a baseline? Can authors comment on this?
The proposed approach is very similar in architecture to [1*, 2*, 3*], which are not discussed/referenced in the paper. Except for pre-training, the only difference from [2] is that copying and generation distributions are softly combined into separate distribution each, independently. Inducing a single output distribution is done instead by deciding which source to use by an MLP layer on decoder state as in Eq. 11. So, I think the authors need to better isolate what the core contribution of this paper is: disentangled decoder or pre-training strategy? If it is the proposed disentangled decoder architecture, then authors should compare with similar architectures [2, 3] in the low-resource setting by initializing encoder and decoder from pre-trained weights on the same Reddit corpus. If it is the pre-training strategy, then it should be compared with various pre-training strategies for sequence generation (e.g., [4*]) proposed recently. For example, it would be useful to include a comparison with fine-tuning a pre-trained MASS  [4*] with the same amount of WoW training data (changing from 1/16 to 1/1).
Presentation of the paper can be improved by 1) changing the name of “document reader” (maybe to “knowledge processor” similar to context) as it essentially attends on the document rather than reading, 2) using abstraction in the technical section to help simplify notation and make it more interpretable.
REFERENCES:
[1*] Get To The Point: Summarization with Pointer-Generator Networks, See et al.
[2*] DeepCopy: Grounded Response Generation with Hierarchical Pointer Networks, Yavuz et al.
[3*] Disentangling Language and Knowledge in Task-Oriented Dialogs, Raghu et al.
[4*] MASS: Masked Sequence to Sequence Pre-training for Language Generation, Song et al.