We introduce kNN-LMs, which extend a pre-trained neural language model (LM) by linearly interpolating it with a k-nearest neighbors (kNN) model. The nearest neighbors are computed according to distance in the pre-trained LM embedding space, and can be drawn from any text collection, including the original LM training data. Applying this augmentation to a strong WIKITEXT-103 LM, with neighbors drawn from the original training set, our kNN-LM achieves a new stateof-the-art perplexity of 15.79 -a 2.9 point improvement with no additional training. We also show that this approach has implications for efficiently scaling up to larger training sets and allows for effective domain adaptation, by simply varying the nearest neighbor datastore, again without further training. Qualitatively, the model is particularly helpful in predicting rare patterns, such as factual knowledge. Together, these results strongly suggest that learning similarity between sequences of text is easier than predicting the next word, and that nearest neighbor search is an effective approach for language modeling in the long tail.
[Overview]
In this paper, the authors proposed a simple but effective way to augmentation the language model through memorization. Specifically, after obtaining a language model on a dataset, the model further uses the dataset to build a lookup table and then a k-nearest neighbor is used to searching the closest tokens for a token during inference. Based on this, the output distribution of a target token during the inference time would be modified accordingly. Through a comprehensive experiments and ablation studies, the authors showed that the proposed strategy can improve the performance of language models significantly for both the in-domain and out-domain testing scenarios. This is very insightful considering recently a lot of language models are focusing on increasing the size of model and training data.
[Pros]:
Overall I think the paper is well-written and presents clearly. Detailed points below:
1. the authors proposed a simple but effective method for increasing the generalization ability of language model through a memorization strategy. Specifically, the authors proposed to build a lookup table which memorizes the representation and output token pairs which are then used for the inference of language model. Different from conventional way, the proposed strategy does not introduce any more parameters in the model and also does not need any more training or fine-tuning on the target dataset.
2. The authors showed that the proposed strategy can improve the performance of language generation model (i.e., transformer) without any extra training or data, as shown in Table 1. Also, using the continuous caches  with KNN-LM further improve the performance.
3. Besides the main results shown in Table 1 and Table 2, the authors also showed using kNN-LM can probably outperforms the model which is directly trained on it. Also, it also supports domain adaptation from one language domain to another domain.
4. Finally, the authors presented a number of ablation studies to investigate how the performance is affected by the method of building datastore, including the size of nearest neighbor, the interpolation parameter, etc. These results are also insightful and meaningful for the readers to understand the method.
[Cons]:
I think this paper is a solid paper. So I would have some suggestions below:
1. The first concern about the method is the efficiency. At page 3, the authors mentioned that the proposed strategy will bring more time cost. It would be good if the authors can perform more systematical analysis on the time cost of building the datastore and inference for the proposed model. 
2. Second, the authors should not only evaluate the proposed method based on transformers. It would be good to test on various language models to verify the generalization ability across different models, including the old-fashioned one like RNN and CNN.
3. Also, the authors should try to extend the proposed model to other language tasks, such as translation.
[Summary]
In this paper, the authors introduced a simple but effective method to augment the pertained language model through memorizations. Though this is not absolutely new and relatively simple , the authors successfully demonstrate that it can be applied to improve the generation of language model much. The. thorough ablation studies help to understand the property of the proposed strategy. I think this paper overall is insightful and thoughtful. It would be good to see the authors add more analysis on the computational complexity and also evaluate on more type of language models.