Pointer Sentinel Mixture Models
Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and corpora we also introduce the freely available WikiText corpus.
This work is basically a combined pointer network applied on language modelling. The smart point is that this paper aims at language modelling with longer context, where a memory of seen words (especially the rare words) would be very useful for predicting the rest of the sentences. Hence, a combination of a pointer network and a standard language model would balance the copying seen words and predicting unseen words. Generally, such as the combined pointer networks applied in sentence compression, a vector representation of the source sequence would be used to compute the gate. This paper, instead, introduces a sentinel vector to carry out the mixture model, which is suitable in the case of language modelling. I would be interested in the variations of sentinel mixture implementation, though the current version has achieved very good results. In addition, the new WikiText language modelling dataset is very interesting. It probably can be a more standard dataset for evaluating the continuously-updated language model benchmarks than ptb dataset. Overall, this is a well-written paper. I recommend it to be accepted.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Pointer Sentinel Mixture Models
Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and corpora we also introduce the freely available WikiText corpus.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work is basically a combined pointer network applied on language modelling. The smart point is that this paper aims at language modelling with longer context, where a memory of seen words (especially the rare words) would be very useful for predicting the rest of the sentences. Hence, a combination of a pointer network and a standard language model would balance the copying seen words and predicting unseen words. Generally, such as the combined pointer networks applied in sentence compression, a vector representation of the source sequence would be used to compute the gate. This paper, instead, introduces a sentinel vector to carry out the mixture model, which is suitable in the case of language modelling. I would be interested in the variations of sentinel mixture implementation, though the current version has achieved very good results. In addition, the new WikiText language modelling dataset is very interesting. It probably can be a more standard dataset for evaluating the continuously-updated language model benchmarks than ptb dataset. Overall, this is a well-written paper. I recommend it to be accepted.
assistant
This work is basically a combined pointer network applied on language modelling. The smart point is that this paper aims at language modelling with longer context, where a memory of seen words (especially the rare words) would be very useful for predicting the rest of the sentences. Hence, a combination of a pointer network and a standard language model would balance the copying seen words and predicting unseen words. Generally, such as the combined pointer networks applied in sentence compression, a vector representation of the source sequence would be used to compute the gate. This paper, instead, introduces a sentinel vector to carry out the mixture model, which is suitable in the case of language modelling. I would be interested in the variations of sentinel mixture implementation, though the current version has achieved very good results. In addition, the new WikiText language modelling dataset is very interesting. It probably can be a more standard dataset for evaluating the continuously-updated language model benchmarks than ptb dataset. Overall, this is a well-written paper. I recommend it to be accepted.