This work introduces a simple network for producing character aware word embeddings. Position agnostic and position aware character embeddings are combined to produce an embedding vector for each word. The learned word representations are shown to be very sparse and facilitate improved results on language modeling tasks, despite using markedly fewer parameters, and without the need to apply dropout. A final experiment suggests that weight sharing contributes to sparsity, increases performance, and prevents overfitting.
This paper presents a new model for composing representations of characters into word embeddings. The starting point of their argument is to include position-specific embeddings of characters rather than just position-independent characters. By adding together position-specific vectors, reasonable results are obtained.
This is an interesting result, but I have a few recommendations to improve the paper.
1) It is a bit hard to assess since it is not evaluated on a standard datasets. There are a number standard datasets for open vocabulary language modeling. E.g., the MWC corpus (http://k-kawakami.com/research/mwc), or even the Penn Treebank (although it is conventionally modeled in closed vocabulary form).
2) There are many existing models for composing characters into words. In addition to those cited in the paper, see the citations listed below. Comparison with those is crucial in a paper like this.
3) Since the predictions are done at the word type level, it is unclear how vocabulary set of the corpus is determined, and what is done with OOV word types at test time (while it is possible to condition on them using the technique in the paper, it is not possible to use this technique for generation).
4) The analysis is interesting, but a more intuitive explanation would be to show nearest neighbor plots.
Some missing citations:
Composing characters into words:
dos Santos and Zadrozny. (2014 ICML) http://proceedings.mlr.press/v32/santos14.pdf
Ling et al. (2015 EMNLP) Finding Function in Form. https://arxiv.org/abs/1508.02096
Additionally, using explicit positional features in modeling language has been used:
Vaswani et al. (2017) Attention is all you need https://arxiv.org/abs/1706.03762
and a variety of other sources.