Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of realvalued vectors. By learning distributed vector representations for words, each training sample informs the neural network model about a combinatorial number of other patterns. In this paper, we exploit the sparsity in natural language even further by encoding each unique input word using a fixed sparse random representation. These sparse codes are then projected onto a smaller embedding space which allows for the encoding of word occurrences from a possibly unknown vocabulary, along with the creation of more compact language models using a reduced number of parameters. We investigate the properties of our encoding mechanism empirically, by evaluating its performance on the widely used Penn Treebank corpus. We show that guaranteeing approximately equidistant vector representations for unique discrete inputs is enough to provide the neural network model with enough information to learn -and make use-of distributed representations for these inputs.
This paper studied a random projection of word embeddings in neural language modeling. Instead of having |V| x m embeddings, the author(s) represented a word with a random, sparse, linear combination {1, 0, -1} of k vector of size m. The experiment on PTB dataset showed that k had to be somewhat close to |V| in order to achieve the comparable perplexity to a feed-forward NLM.
Overall, I am not sure what we could gain from this research direction. The advantage of this random encoding was to reduce the number of parameters for an embedding layer, but the results showed we gained much PPL from a 25% reduction in embedding size (Table 1). In addition, the fact that the random projections preserved the inner product (centered at zero) was probably not desirable. It might be more fruitful if these linear combinations were learned or sub-senses of words (e.g. [1]).
The experiments were quite extensive on the hyper-parameters and showed how the models performed under different settings. However, these were done using 1 dataset and also a simple feed-forward network (rather than LSTM). I can understand the point that training NNLM accelerates the experiments, but the author(s) should consider trying a simply LSTM model after the best settings had been discovered (e.g. Table 1). PTB also has a very unnatural vocabulary distribution as pointed out in [2]. Thus, it might be helpful to test the result on another dataset (e.g. WikiText).
Other comments
1. I do not get the point of bringing up NCE. Did you actually use NCE loss? Did you only refer to NCE as a weight tying which can be used in a standard XENT loss [3]? The first paragraph of 3.3 did not help clarify this point either.
2. In Figure 3, the baseline got different perplexity between 3(a) and 3(b). 
3. Shouldn't random indexing produce non-uniform numbers of non-zero entries depending on alpha? Why did you have an exact number of non-zero entries, s, in the experiments?
3. Some typos
- "... is that instead of trying to probability ..." => "... tying ..."
- "... All models sare trained ..." => "... are ..."
- "... Tho get the feature ..." => ?
References
[1] S. Arora et al., 2016. Linear Algebraic Structure of Word Senses, with Applications to Polysemy
[2] S. Merity et al., 2016. Pointer Sentinel Mixture Models
[3] Y. Gal et al., 2015. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks