Riemannian Optimization for Skip-Gram Negative Sampling
Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in *word2vec* software, is usually optimized by stochastic gradient descent. It can be shown that optimizing for SGNS objective can be viewed as an optimization problem of searching for a good matrix with the low-rank constraint. The most standard way to solve this type of problems is to apply Riemannian optimization framework to optimize the SGNS objective over the manifold of required low-rank matrices. In this paper, we propose an algorithm that optimizes SGNS objective using Riemannian optimization and demonstrates its superiority over popular competitors, such as the original method to train SGNS and SVD over SPPMI matrix.
Dear authors, The authors* response clarified some of my confusion. But I still have the following question: -- The response said a first contribution is a different formulation: you divide the word embedding learning into two steps, step 1 looks for a low-rank X (by Riemannian optimization), step 2 factorizes X into two matrices (W, C). You are claiming that your model outperforms previous approaches that directly optimizes over (W, C). But since the end result (the factors) is the same, can the authors provide some intuition and justification why the proposed method works better? As far as I can see, though parameterized differently, the first step of your method and previous methods (SGD) are both optimizing over low-rank matrices. Admittedly, Riemannian optimization avoids the rotational degree of freedom (the invertible matrix S you are mentioning in sec 2.3), but I am not 100% certain at this point this is the source of your gain; learning curves of objectives would help to see if Riemannian optimization is indeed more effective. -- Another detail I could not easily find is the following. You said a disadvantage of other approaches is that their factors W and C do not directly reflect similarity. Did you try to multiply the factors W and C from other optimizers and then factorize the product using the method in section 2.3, and use the new W for your downstream tasks? I am not sure if this would cause much difference in the performance. Overall, I think it is always interesting to apply advanced optimization techniques to machine learning problems. The current paper would be stronger from the machine learning perspective, if more thorough comparison and discussion (as mentioned above) are provided. On the other hand, my expertise is not in NLP and I leave it to other reviewers to decide the significance in experimental results.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Riemannian Optimization for Skip-Gram Negative Sampling
Skip-Gram Negative Sampling (SGNS) word embedding model, well known by its implementation in *word2vec* software, is usually optimized by stochastic gradient descent. It can be shown that optimizing for SGNS objective can be viewed as an optimization problem of searching for a good matrix with the low-rank constraint. The most standard way to solve this type of problems is to apply Riemannian optimization framework to optimize the SGNS objective over the manifold of required low-rank matrices. In this paper, we propose an algorithm that optimizes SGNS objective using Riemannian optimization and demonstrates its superiority over popular competitors, such as the original method to train SGNS and SVD over SPPMI matrix.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



Dear authors, The authors* response clarified some of my confusion. But I still have the following question: -- The response said a first contribution is a different formulation: you divide the word embedding learning into two steps, step 1 looks for a low-rank X (by Riemannian optimization), step 2 factorizes X into two matrices (W, C). You are claiming that your model outperforms previous approaches that directly optimizes over (W, C). But since the end result (the factors) is the same, can the authors provide some intuition and justification why the proposed method works better? As far as I can see, though parameterized differently, the first step of your method and previous methods (SGD) are both optimizing over low-rank matrices. Admittedly, Riemannian optimization avoids the rotational degree of freedom (the invertible matrix S you are mentioning in sec 2.3), but I am not 100% certain at this point this is the source of your gain; learning curves of objectives would help to see if Riemannian optimization is indeed more effective. -- Another detail I could not easily find is the following. You said a disadvantage of other approaches is that their factors W and C do not directly reflect similarity. Did you try to multiply the factors W and C from other optimizers and then factorize the product using the method in section 2.3, and use the new W for your downstream tasks? I am not sure if this would cause much difference in the performance. Overall, I think it is always interesting to apply advanced optimization techniques to machine learning problems. The current paper would be stronger from the machine learning perspective, if more thorough comparison and discussion (as mentioned above) are provided. On the other hand, my expertise is not in NLP and I leave it to other reviewers to decide the significance in experimental results.
assistant
Dear authors, The authors* response clarified some of my confusion. But I still have the following question: -- The response said a first contribution is a different formulation: you divide the word embedding learning into two steps, step 1 looks for a low-rank X (by Riemannian optimization), step 2 factorizes X into two matrices (W, C). You are claiming that your model outperforms previous approaches that directly optimizes over (W, C). But since the end result (the factors) is the same, can the authors provide some intuition and justification why the proposed method works better? As far as I can see, though parameterized differently, the first step of your method and previous methods (SGD) are both optimizing over low-rank matrices. Admittedly, Riemannian optimization avoids the rotational degree of freedom (the invertible matrix S you are mentioning in sec 2.3), but I am not 100% certain at this point this is the source of your gain; learning curves of objectives would help to see if Riemannian optimization is indeed more effective. -- Another detail I could not easily find is the following. You said a disadvantage of other approaches is that their factors W and C do not directly reflect similarity. Did you try to multiply the factors W and C from other optimizers and then factorize the product using the method in section 2.3, and use the new W for your downstream tasks? I am not sure if this would cause much difference in the performance. Overall, I think it is always interesting to apply advanced optimization techniques to machine learning problems. The current paper would be stronger from the machine learning perspective, if more thorough comparison and discussion (as mentioned above) are provided. On the other hand, my expertise is not in NLP and I leave it to other reviewers to decide the significance in experimental results.