Various machine learning tasks can benefit from access to external information of different modalities, such as text and images. Recent work has focused on learning architectures with large memories capable of storing this knowledge. We propose augmenting generative Transformer neural networks with KNN-based Information Fetching (KIF) modules. Each KIF module learns a read operation to access fixed external knowledge. We apply these modules to generative dialogue modeling, a challenging task where information must be flexibly retrieved and incorporated to maintain the topic and flow of conversation. We demonstrate the effectiveness of our approach by identifying relevant knowledge from Wikipedia, images, and human-written dialogue utterances, and show that leveraging this retrieved information improves model performance, measured by automatic and human evaluation.
### Summary
​
This paper provides a framework to augment dialogue generation with external data sources using K-Nearest Neighbors in the embedding space. The idea seems simple and intuitive, and the results show improvements over prior work in dialogue generation and retrieval.
​
​
### Strengths
- The paper shows quantitative improvement over some prior works on dialogue agents (however this needs to be correctly validated).
- The reviewer appreciates the human study and conversation example provided in the paper to qualitatively evaluate their model.
- The paper provides ablation studies of the various training tricks for each dataset they have proposed.
​
### Weaknesses
- There are certain changes in the training pipeline that makes the comparison with prior work difficult (Tables 1 and 2), and find the real contribution of data-augmentation caused by KNN-based information fetching (KIF). e.g., In Wizard of Wikipedia experiment, most recent dialogue utterance and turn number are used as salient features. Similarly, the paper utilizes a "personality" feature in ImageChat dataset. It seems the results taken from prior papers have different training settings. Could the authors verify that the extra assumptions made for their model are equivalently applied to other generative baselines? If not, the reviewer recommends to provide some experiments with same conditions.
- Could the authors provide the embedding dimensions and other training details, to assess the contribution of different components of the input's embeddings.
​
​
#### Minor:
- Is there a sound reason behind using concatenation of input representation and fetched representations? This design choice makes the architecture inflexible to the number of data sources. Another way to combine embeddings - e.g. addition or inner product, can also be tried to see if they provide performance improvements.
- It is mentioned that attention based modules scale poorly with large sized datasets. If the authors conducted a quantitative evaluation to test this, it would be valuable to add it to the paper.
- The title and abstract should be limited to generative dialogue modeling, instead of transformers, since the contributions proposed are more suited for this particular application and are quite engineered. Hence, it is not correct to make this claim for transformers in general.
​
​
### Score
Weak Reject (Leaning towards Accept if appropriate experiements can be provided)