Partially Mutual Exclusive Softmax for Positive and Unlabeled data
In recent years, softmax together with its fast approximations has become the de-facto loss function for deep neural networks with multiclass predictions. However, softmax is used in many problems that do not fully fit the multiclass framework and where the softmax assumption of mutually exclusive outcomes can lead to biased results. This is often the case for applications such as language modeling, next event prediction and matrix factorization, where many of the potential outcomes are not mutually exclusive, but are more likely to be independent conditionally on the state. To this end, for the set of problems with positive and unlabeled data, we propose a relaxation of the original softmax formulation, where, given the observed state, each of the outcomes are conditionally independent but share a common set of negatives. Since we operate in a regime where explicit negatives are missing, we create an adversarially-trained model of negatives and derive a new negative sampling and weighting scheme which we denote as Cooperative Importance Sampling (CIS). We show empirically the advantages of our newly introduced negative sampling scheme by pluging it in the Word2Vec algorithm and benching it extensively against other negative sampling schemes on both language modeling and matrix factorization tasks and show large lifts in performance.
 The mutually exclusive assumption of traditional softmax can be biased in case negative samples are not explicitly defined. This paper presents Cooperative Importance Sampling towards resolving this problem. The authors experimentally verify the effectiveness of the proposed approach using different tasks including applying matrix factorization in recommender system, language modeling tasks and a task on synthetic data. I like this interesting idea, and I agree with the authors that softmax does exist certain problem especially when negative samples are not well defined. I appreciate the motivation of this work from the PU learning setting. It would be interested to show more results in PU learning setting using some synthetic data. I am interested to see the benefit of this extension of softmax with respect to different amount of labeled positive samples. However, I am not completely convinced that the proposed method would be a necessary choice for language modeling tasks. --To me, the proposed method has close connection to 2-gram language model. --But for language tasks, and other sequential input, we typically make prediction based on representation of very large context. Let’s say, we would like to make prediction for time step t given the context of word_{1:t} based on some recurrent model, do you think the proposed softmax can generally bring sizable improvement with respect to traditional choices. And how? By the way, I think the proposed method would also be applicable in the soft-label setting. For the experiments part, maybe put more details and discussions to the supplementary material. A few concrete questions. -- In some tables and settings, you only look at prec@1, why? I expect the proposed approach would work better in prec@K. -- Can you provide more concrete analysis fortable 6? Why proposed methods does not work well for syntactic. -- Describe a little bit details about MF techniques and hyper-parameters you used.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Partially Mutual Exclusive Softmax for Positive and Unlabeled data
In recent years, softmax together with its fast approximations has become the de-facto loss function for deep neural networks with multiclass predictions. However, softmax is used in many problems that do not fully fit the multiclass framework and where the softmax assumption of mutually exclusive outcomes can lead to biased results. This is often the case for applications such as language modeling, next event prediction and matrix factorization, where many of the potential outcomes are not mutually exclusive, but are more likely to be independent conditionally on the state. To this end, for the set of problems with positive and unlabeled data, we propose a relaxation of the original softmax formulation, where, given the observed state, each of the outcomes are conditionally independent but share a common set of negatives. Since we operate in a regime where explicit negatives are missing, we create an adversarially-trained model of negatives and derive a new negative sampling and weighting scheme which we denote as Cooperative Importance Sampling (CIS). We show empirically the advantages of our newly introduced negative sampling scheme by pluging it in the Word2Vec algorithm and benching it extensively against other negative sampling schemes on both language modeling and matrix factorization tasks and show large lifts in performance.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The mutually exclusive assumption of traditional softmax can be biased in case negative samples are not explicitly defined. This paper presents Cooperative Importance Sampling towards resolving this problem. The authors experimentally verify the effectiveness of the proposed approach using different tasks including applying matrix factorization in recommender system, language modeling tasks and a task on synthetic data. I like this interesting idea, and I agree with the authors that softmax does exist certain problem especially when negative samples are not well defined. I appreciate the motivation of this work from the PU learning setting. It would be interested to show more results in PU learning setting using some synthetic data. I am interested to see the benefit of this extension of softmax with respect to different amount of labeled positive samples. However, I am not completely convinced that the proposed method would be a necessary choice for language modeling tasks. --To me, the proposed method has close connection to 2-gram language model. --But for language tasks, and other sequential input, we typically make prediction based on representation of very large context. Let’s say, we would like to make prediction for time step t given the context of word_{1:t} based on some recurrent model, do you think the proposed softmax can generally bring sizable improvement with respect to traditional choices. And how? By the way, I think the proposed method would also be applicable in the soft-label setting. For the experiments part, maybe put more details and discussions to the supplementary material. A few concrete questions. -- In some tables and settings, you only look at prec@1, why? I expect the proposed approach would work better in prec@K. -- Can you provide more concrete analysis fortable 6? Why proposed methods does not work well for syntactic. -- Describe a little bit details about MF techniques and hyper-parameters you used.
assistant
 The mutually exclusive assumption of traditional softmax can be biased in case negative samples are not explicitly defined. This paper presents Cooperative Importance Sampling towards resolving this problem. The authors experimentally verify the effectiveness of the proposed approach using different tasks including applying matrix factorization in recommender system, language modeling tasks and a task on synthetic data. I like this interesting idea, and I agree with the authors that softmax does exist certain problem especially when negative samples are not well defined. I appreciate the motivation of this work from the PU learning setting. It would be interested to show more results in PU learning setting using some synthetic data. I am interested to see the benefit of this extension of softmax with respect to different amount of labeled positive samples. However, I am not completely convinced that the proposed method would be a necessary choice for language modeling tasks. --To me, the proposed method has close connection to 2-gram language model. --But for language tasks, and other sequential input, we typically make prediction based on representation of very large context. Let’s say, we would like to make prediction for time step t given the context of word_{1:t} based on some recurrent model, do you think the proposed softmax can generally bring sizable improvement with respect to traditional choices. And how? By the way, I think the proposed method would also be applicable in the soft-label setting. For the experiments part, maybe put more details and discussions to the supplementary material. A few concrete questions. -- In some tables and settings, you only look at prec@1, why? I expect the proposed approach would work better in prec@K. -- Can you provide more concrete analysis fortable 6? Why proposed methods does not work well for syntactic. -- Describe a little bit details about MF techniques and hyper-parameters you used.