A Matrix Approximation View of NCE that Justifies Self-Normalization
Self-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. This property is useful to computationally-intensive neural network classifiers, as the cost of computing the partition function grows linearly with the number of classes and may become prohibitive. In particular, since neural language models may deal with up to millions of classes, their self-normalization properties received notable attention. Several recent studies empirically found that language models, trained using Noise Contrastive Estimation (NCE), exhibit self-normalization, but could not explain why. In this study, we provide a theoretical justification to this property by viewing NCE as a low-rank matrix approximation. Our empirical investigation compares NCE to the alternative explicit approach for self-normalizing language models. It also uncovers a surprising negative correlation between self-normalization and perplexity, as well as some regularity in the observed errors that may potentially be used for improving self-normalization algorithms in the future.
 The paper presents a proof of the self normalization of NCE as a result of being a low-rank matrix approximation of low-rank approximation of the normalized conditional probabilities matrix. However, it seems that in equation 4, the authors assume that the noise distribution is a unigram model over words. However, one is allowed to use any noise distribution in NCE, and convergence should be quicker with those distributions that are close to the true distribution. Does the argument hold for general noise distributions ? With this assumption, they can borrow easily from Goldberg and Levy, 2014 for the proof. In experiments, they find that while NCE does result in self-normalization, it is inversely correlated with perplexity which is a bit surprising. The paper is interesting but lacks strong empirical results. It could be stronger if they could exploit some of their findings to improve language modeling over a strong baseline.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A Matrix Approximation View of NCE that Justifies Self-Normalization
Self-normalizing discriminative models approximate the normalized probability of a class without having to compute the partition function. This property is useful to computationally-intensive neural network classifiers, as the cost of computing the partition function grows linearly with the number of classes and may become prohibitive. In particular, since neural language models may deal with up to millions of classes, their self-normalization properties received notable attention. Several recent studies empirically found that language models, trained using Noise Contrastive Estimation (NCE), exhibit self-normalization, but could not explain why. In this study, we provide a theoretical justification to this property by viewing NCE as a low-rank matrix approximation. Our empirical investigation compares NCE to the alternative explicit approach for self-normalizing language models. It also uncovers a surprising negative correlation between self-normalization and perplexity, as well as some regularity in the observed errors that may potentially be used for improving self-normalization algorithms in the future.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper presents a proof of the self normalization of NCE as a result of being a low-rank matrix approximation of low-rank approximation of the normalized conditional probabilities matrix. However, it seems that in equation 4, the authors assume that the noise distribution is a unigram model over words. However, one is allowed to use any noise distribution in NCE, and convergence should be quicker with those distributions that are close to the true distribution. Does the argument hold for general noise distributions ? With this assumption, they can borrow easily from Goldberg and Levy, 2014 for the proof. In experiments, they find that while NCE does result in self-normalization, it is inversely correlated with perplexity which is a bit surprising. The paper is interesting but lacks strong empirical results. It could be stronger if they could exploit some of their findings to improve language modeling over a strong baseline.
assistant
 The paper presents a proof of the self normalization of NCE as a result of being a low-rank matrix approximation of low-rank approximation of the normalized conditional probabilities matrix. However, it seems that in equation 4, the authors assume that the noise distribution is a unigram model over words. However, one is allowed to use any noise distribution in NCE, and convergence should be quicker with those distributions that are close to the true distribution. Does the argument hold for general noise distributions ? With this assumption, they can borrow easily from Goldberg and Levy, 2014 for the proof. In experiments, they find that while NCE does result in self-normalization, it is inversely correlated with perplexity which is a bit surprising. The paper is interesting but lacks strong empirical results. It could be stronger if they could exploit some of their findings to improve language modeling over a strong baseline.