Neural Random Projections for Language Modelling
Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of real-valued vectors. By learning distributed vector representations for words, each training sample informs the neural network model about a combinatorial number of other patterns. In this paper, we exploit the sparsity in natural language even further by encoding each unique input word using a fixed sparse random representation. These sparse codes are then projected onto a smaller embedding space which allows for the encoding of word occurrences from a possibly unknown vocabulary, along with the creation of more compact language models using a reduced number of parameters. We investigate the properties of our encoding mechanism empirically, by evaluating its performance on the widely used Penn Treebank corpus. We show that guaranteeing approximately equidistant vector representations for unique discrete inputs is enough to provide the neural network model with enough information to learn --and make use-- of distributed representations for these inputs.
 This paper presents some experiments using random projections instead of embeddings from a 1-of-V encoding. Experiments on the Penn TreeBank benchmark data set show that in a feed-forward language modeling architecture similar to that of (Bengio, 2003), the random projections substantially reduce the number of parameters of the model while not harming perplexity too much. The paper would need to be improved substantially in order to appear at a conference like ICLR. First, the novelty of the approach is limited -- the approach amounts to using a sparse integer layer instead of a floating-point layer within a feed-forward architecture. Second, and more importantly, the experiments need to be re-done to better measure the practical impact of the techniques. First, larger data sets such as Wikitext-2 and Wikitext-103, and/or the billion-word benchmark, are needed to understand how well the approach works in practical LM settings. Second, the paper needs to use more state-of-the-art architectures. Language modeling is a fast-moving field, so the very latest and greatest techniques are not strictly necessary for this paper, but at least midsize LSTM models that get scores in the ~80 ppl range for Penn Treebank are important, otherwise it becomes very questionable whether the results will provide any practical impact in today*s best models. Finally, the paper needs to compare its parameter-reduction approaches against other compression and hyperparameter optimization techniques. Changing the number/sizes of the network layers or using sparse weight matrices (perhaps with sparsity-inducing regularization) would be natural ways to reduce the parameter space. In my opinion, due to how many researchers are and have been looking into improvements of language modeling, the authors may find it hard to break new ground in this direction. Minor In the start of Section 3, it is not clear why having the projection be sparse is desired. Later, space (and time) efficiency is revealed as the motivation for the sparsity, but it would be helpful if the paper said this earlier. Equation 6 seems to have an error, the probability should be P(w_t | w_t-1...) instead of P(w_t , w_t-1...) if this is to represent the standard LM objective (the probability of the corpus). Sec 3.3: "all models sare"
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Neural Random Projections for Language Modelling
Neural network-based language models deal with data sparsity problems by mapping the large discrete space of words into a smaller continuous space of real-valued vectors. By learning distributed vector representations for words, each training sample informs the neural network model about a combinatorial number of other patterns. In this paper, we exploit the sparsity in natural language even further by encoding each unique input word using a fixed sparse random representation. These sparse codes are then projected onto a smaller embedding space which allows for the encoding of word occurrences from a possibly unknown vocabulary, along with the creation of more compact language models using a reduced number of parameters. We investigate the properties of our encoding mechanism empirically, by evaluating its performance on the widely used Penn Treebank corpus. We show that guaranteeing approximately equidistant vector representations for unique discrete inputs is enough to provide the neural network model with enough information to learn --and make use-- of distributed representations for these inputs.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 This paper presents some experiments using random projections instead of embeddings from a 1-of-V encoding. Experiments on the Penn TreeBank benchmark data set show that in a feed-forward language modeling architecture similar to that of (Bengio, 2003), the random projections substantially reduce the number of parameters of the model while not harming perplexity too much. The paper would need to be improved substantially in order to appear at a conference like ICLR. First, the novelty of the approach is limited -- the approach amounts to using a sparse integer layer instead of a floating-point layer within a feed-forward architecture. Second, and more importantly, the experiments need to be re-done to better measure the practical impact of the techniques. First, larger data sets such as Wikitext-2 and Wikitext-103, and/or the billion-word benchmark, are needed to understand how well the approach works in practical LM settings. Second, the paper needs to use more state-of-the-art architectures. Language modeling is a fast-moving field, so the very latest and greatest techniques are not strictly necessary for this paper, but at least midsize LSTM models that get scores in the ~80 ppl range for Penn Treebank are important, otherwise it becomes very questionable whether the results will provide any practical impact in today*s best models. Finally, the paper needs to compare its parameter-reduction approaches against other compression and hyperparameter optimization techniques. Changing the number/sizes of the network layers or using sparse weight matrices (perhaps with sparsity-inducing regularization) would be natural ways to reduce the parameter space. In my opinion, due to how many researchers are and have been looking into improvements of language modeling, the authors may find it hard to break new ground in this direction. Minor In the start of Section 3, it is not clear why having the projection be sparse is desired. Later, space (and time) efficiency is revealed as the motivation for the sparsity, but it would be helpful if the paper said this earlier. Equation 6 seems to have an error, the probability should be P(w_t | w_t-1...) instead of P(w_t , w_t-1...) if this is to represent the standard LM objective (the probability of the corpus). Sec 3.3: "all models sare"
assistant
 This paper presents some experiments using random projections instead of embeddings from a 1-of-V encoding. Experiments on the Penn TreeBank benchmark data set show that in a feed-forward language modeling architecture similar to that of (Bengio, 2003), the random projections substantially reduce the number of parameters of the model while not harming perplexity too much. The paper would need to be improved substantially in order to appear at a conference like ICLR. First, the novelty of the approach is limited -- the approach amounts to using a sparse integer layer instead of a floating-point layer within a feed-forward architecture. Second, and more importantly, the experiments need to be re-done to better measure the practical impact of the techniques. First, larger data sets such as Wikitext-2 and Wikitext-103, and/or the billion-word benchmark, are needed to understand how well the approach works in practical LM settings. Second, the paper needs to use more state-of-the-art architectures. Language modeling is a fast-moving field, so the very latest and greatest techniques are not strictly necessary for this paper, but at least midsize LSTM models that get scores in the ~80 ppl range for Penn Treebank are important, otherwise it becomes very questionable whether the results will provide any practical impact in today*s best models. Finally, the paper needs to compare its parameter-reduction approaches against other compression and hyperparameter optimization techniques. Changing the number/sizes of the network layers or using sparse weight matrices (perhaps with sparsity-inducing regularization) would be natural ways to reduce the parameter space. In my opinion, due to how many researchers are and have been looking into improvements of language modeling, the authors may find it hard to break new ground in this direction. Minor In the start of Section 3, it is not clear why having the projection be sparse is desired. Later, space (and time) efficiency is revealed as the motivation for the sparsity, but it would be helpful if the paper said this earlier. Equation 6 seems to have an error, the probability should be P(w_t | w_t-1...) instead of P(w_t , w_t-1...) if this is to represent the standard LM objective (the probability of the corpus). Sec 3.3: "all models sare"