Standard deep learning systems require thousands or millions of examples to learn a concept, and cannot integrate new concepts easily. By contrast, humans have an incredible ability to do one-shot or few-shot learning. For instance, from just hearing a word used in a sentence, humans can infer a great deal about it, by leveraging what the syntax and semantics of the surrounding words tells us. Here, we draw inspiration from this to highlight a simple technique by which deep recurrent networks can similarly exploit their prior knowledge to learn a useful representation for a new word from little data. This could make natural language processing systems much more flexible, by allowing them to learn continually from the new words they encounter.
Paper Summary
From just seeing a word used in a sentence, humans can infer a lot about this word by leveraging the surrounding words. Based on this idea, this work tries to obtain a better understanding of words in the one-shot or few-shot setting by leveraging surrounding word. They do this by language modeling sentences which contain rarely seen or never seen words. They evaluated their model using percent change in perplexity on test sentences containing new word by varying the number of training sentences containing this word. 3 Proposed Methods to model few-shot words: (1) beginning with random embedding, (2) beginning with zero embedding (3) beginning with the centroid of other words in the sentence. They compare to 2 Baseline Methods: (1) centroid of other words in the sentence, and (2) full training including the sparse words. Their results show that learning from centroids of other words can outperform full training on the new words. 
Explanation
The paper is well written, and the experiments are well explained.  It is an interesting paper, and a research topic which is not well studied. The experiments are reasonable. The method seems to work well. 
However, the method provides a very marginal difference between the previous method in Lazaridou et al. (2017). They just use backdrop to learn from this starting position. The main contribution of this work is the evaluation section. 
Why only use the PTB language modeling task. Why not use the task in Gauthier & Mordatch or Hermann et al. The one task of language modeling shows promising results, but it’s not totally convincing. 
One of the biggest caveats is that the experiments are only done in a few words. I’m not sure why more couldn’t have been done. This is discussed in section 4.1, but I think some of these differences could have been alleviated if there were more experiments done. Regardless, the experiments on the 8 words that they did chose were well done. 
I don’t think that section 3.3 (embedding similarity) is particularly useful.