Word embedding is an essential building block for deep learning methods for natural language processing. Although word embedding has been extensively studied over the years, the problem of how to effectively embed numerals, a special subset of words, is still underexplored. Existing word embedding methods do not learn numeral embeddings well because there are an infinite number of numerals and their individual appearances in training corpora are highly scarce. In this paper, we propose two novel numeral embedding methods that can handle the out-ofvocabulary (OOV) problem for numerals. We first induce a finite set of prototype numerals using either a self-organizing map or a Gaussian mixture model. We then represent the embedding of a numeral as a weighted average of the prototype number embeddings. Numeral embeddings represented in this manner can be plugged into existing word embedding learning approaches such as skip-gram for training. We evaluated our methods and showed its effectiveness on four intrinsic and extrinsic tasks: word similarity, embedding numeracy, numeral prediction, and sequence labeling.
The paper proposes a novel method for embedding numerals which can be learned by using neural word embedding learning techniques. The paper motivates the work by reviewing the difficulty of embedding components to represent numerals: OOV in most cases. Their main contribution is the introduction of a method composes numeral embedding by a weighted average of prototype embeddings based on the similarities between the numeral and prototypes. There are two proposed prototypes: SOM and GMM and the similarity functions are an absolute difference and the density function respectively. During the training, the numerals have the proposed embeddings while the others have normal word embeddings. The paper slightly modifies the negative sampling to ensure numerals being sampled. A series of 4 empirical studies have been presented. First, the paper confirms that the proposed method does not negatively affect non-numeral embeddings. And then, the quality of the numeral embeddings are evaluated and compared. The experiments show that the proposed method has better performance on numerical property tests, numeral prediction, and a sequence labeling task.
Overall, this paper has a novel contribution. The proposed method is well motivated and quite justified by the experiments abliet lacking comparison with previous published results. However, it has some weaknesses.
For the method part, one of the limitations is that it is not an end-to-end method and requires a regular expression to identify the numeral. Second, the weighted average of the prototypes is reasonable, but the similarity function only relies on the magnitude. I think there are other aspects of numerical tokens that it might not be able to capture (e.g. “2019” is similar to “19” in some context). In terms of training, I think it is not hard to extend the method to full language model training (using softmax). However, adding all numerals to the vocabulary would add significant overhead.
For the experiments, some design decisions are left unjustified. For example, a simple ablation experiment on the squashing function will be helpful. Furthermore, guidelines or empirical results on the effect of the number of prototypes can increase the impact of the paper. Finally, I think an analysis of the performance of numerical types will be helpful for future works (e.g., dates, phone numbers, currencies, etc, or discrete vs continuous). 
Minor comments and questions:
1. The log sigmoid in equation 6 is a bit strange, isn’t it log sum exp(). https://arxiv.org/pdf/1402.3722v1.pdf
2. Why do you create a new dataset for the experiment in section 4.3?
3. In section 4.4, you rank only numerals in the test set, but the scores are computed based on all numerals in the vocab. Do you have the performance of ranking all numerals?
4. Just to confirm the “training” in section 4.4 refers to learning the embedding using the skip-gram model, right?