Words in natural language follow a Zipfian distribution whereby some words are frequent but most are rare. Learning representations for words in the "long tail" of this distribution requires enormous amounts of data. Representations of rare words trained directly on end tasks are usually poor, requiring us to pre-train embeddings on external data, or treat all rare words as out-of-vocabulary words with a unique representation. We provide a method for predicting embeddings of rare words on the fly from small amounts of auxiliary data with a network trained end-to-end for the downstream task. We show that this improves results against baselines where embeddings are trained on the end task for reading comprehension, recognizing textual entailment and language modeling.
This paper examines ways of producing word embeddings for rare words on demand. The key real-world use case is for domain specific terms, but here the techniques are demonstrated on rarer words in standard data sets. The strength of this paper is that it both gives a more systematic framework for and builds on existing ideas (character-based models, using dictionary definitions) to implement them as part of a model trained on the end task.
The contribution is clear but not huge. In general, for the scope of the paper, it seems like what is here could fairly easily have been made into a short paper for other conferences that have that category. The basic method easily fits within 3 pages, and while the presentation of the experiments would need to be much briefer, this seems quite possible. More things could have been considered. Some appear in the paper, and there are some fairly natural other ones such as mining some use contexts of a word (such as just from Google snippets) rather than only using textual definitions from wordnet. The contributions are showing that existing work using character-level models and definitions can be improved by optimizing representation learning in the context of the final task, and the idea of adding a learned linear transformation matrix inside the mean pooling model (p.3). However, it is not made very clear why this matrix is needed or what the qualitative effect of its addition is.
The paper is clearly written. 
A paper that should be referred to is the (short) paper of Dhingra et al. (2017): A Comparative Study of Word Embeddings
for Reading Comprehension https://arxiv.org/pdf/1703.00993.pdf . While it in no way covers the same ground as this paper it is relevant as follows: This paper assumes a baseline that is also described in that paper of using a fixed vocab and mapping other words to UNK. However, they point out that at least for matching tasks like QA and NLI that one can do better by assigning random vectors on the fly to unknown words. That method could also be considered as a possible approach to compare against here.
Other comments:
 - The paper suggests a couple of times including at the end of the 2nd Intro paragraph that you can't really expect spelling models to perform well in representing the semantics of arbitrary words (which are not morphological derivations, etc.). While this argument has intuitive appeal, it seems to fly in the face of the fact that actually spelling models, including in this paper, seem to do surprisingly well at learning such arbitrary semantics.
 - p.2: You use pretrained GloVe vectors that you do not update. My impression is that people have had mixed results, sometimes better, sometimes worse with updating pretrained vectors or not. Did you try it both ways?
 - fn. 1: Perhaps slightly exaggerates the point being made, since people usually also get good results with the GloVe or word2vec model trained on "only" 6 billion words – 2 orders of magnitude less data.
 - p.4. When no definition is available, is making e_d(w) a zero vector worse than or about the same as using a trained UNK vector?
 - Table 1: The baseline seems reasonable (near enough to the quality of the original Salesforce model from 2016 (66 F1) but well below current best single models of around 76-78 F1. The difference between D1 and D3 does well illustrate that better definition learning is done with backprop from end objective. This model shows the rather strong performance of spelling models – at least on this task – which he again benefit from training in the context of the end objective. 
 - Fig 2: It's weird that only the +dict (left) model learns to connect "In" and "where". The point made in the text between "Where" and "overseas" is perfectly reasonable, but it is a mystery why the base model on the right doesn't learn to associate the common words "where" and "in" both commonly expressing a location.
 - Table 2: These results are interestingly different. Dict is much more useful than spelling here. I guess that is because of the nature of NLI, but it isn't 100% clear why NLI benefits so much more than QA from definitional knowledge.
 - p.7: I was slightly surprised by how small vocabs (3k and 5k words) are said to be optimal for NLI (and similar remarks hold for SQuAD). My impression is that most papers on NLI use much larger vocabs, no?
 - Fig 3: This could really be drawn considerably better: make the dots bigger and their colors more distinct.
 - Table 3: The differences here are quite small and perhaps the least compelling, but the same trends hold.