Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.
This work offers a theoretical justification for reusing the input word embedding in the output projection layer. It does by proposing an additional loss that is designed to minimize the distance between the predictive distribution and an estimate of the true data distribution. This is a nice setup since it can effectively smooth over the labels given as input. However, the construction of the estimate of the true data distribution seems engineered to provide the weight tying justification in Eqs. 3.6 and 3.7. It is not obvious why the projection matrix L in Eq 3.6 (let*s rename it to L*) should be the same as that in Eq. 2.1. For example, L* could be obtained through word2vec embeddings trained on a large dataset or it could be learned as an additional set of parameters. In the case that L* is a new learned matrix, it seems the result in Eq 4.5 is to use an independent matrix for the output projection layer, as is usually done. The experimental results are good and provide support for the approximate derivation done in section 4, particularly the distance plots in figure 1. Minor comments: Third line in abstract: where model -> where the model Second line in section 7: into space -> into the space Shouldn*t the RHS in Eq 3.5 be sum 	ilde{y_{t,i}}(frac{hat{y}_t}{	ilde{y_{t,i}}} - e_i) ?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This work offers a theoretical justification for reusing the input word embedding in the output projection layer. It does by proposing an additional loss that is designed to minimize the distance between the predictive distribution and an estimate of the true data distribution. This is a nice setup since it can effectively smooth over the labels given as input. However, the construction of the estimate of the true data distribution seems engineered to provide the weight tying justification in Eqs. 3.6 and 3.7. It is not obvious why the projection matrix L in Eq 3.6 (let*s rename it to L*) should be the same as that in Eq. 2.1. For example, L* could be obtained through word2vec embeddings trained on a large dataset or it could be learned as an additional set of parameters. In the case that L* is a new learned matrix, it seems the result in Eq 4.5 is to use an independent matrix for the output projection layer, as is usually done. The experimental results are good and provide support for the approximate derivation done in section 4, particularly the distance plots in figure 1. Minor comments: Third line in abstract: where model -> where the model Second line in section 7: into space -> into the space Shouldn*t the RHS in Eq 3.5 be sum 	ilde{y_{t,i}}(frac{hat{y}_t}{	ilde{y_{t,i}}} - e_i) ?
assistant
This work offers a theoretical justification for reusing the input word embedding in the output projection layer. It does by proposing an additional loss that is designed to minimize the distance between the predictive distribution and an estimate of the true data distribution. This is a nice setup since it can effectively smooth over the labels given as input. However, the construction of the estimate of the true data distribution seems engineered to provide the weight tying justification in Eqs. 3.6 and 3.7. It is not obvious why the projection matrix L in Eq 3.6 (let*s rename it to L*) should be the same as that in Eq. 2.1. For example, L* could be obtained through word2vec embeddings trained on a large dataset or it could be learned as an additional set of parameters. In the case that L* is a new learned matrix, it seems the result in Eq 4.5 is to use an independent matrix for the output projection layer, as is usually done. The experimental results are good and provide support for the approximate derivation done in section 4, particularly the distance plots in figure 1. Minor comments: Third line in abstract: where model -> where the model Second line in section 7: into space -> into the space Shouldn*t the RHS in Eq 3.5 be sum 	ilde{y_{t,i}}(frac{hat{y}_t}{	ilde{y_{t,i}}} - e_i) ?