Answers.Body: <p>The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a given input can either belong or not. More specifically, what you're trying to do is multi-label classification, which is different from multi-class classification. The difference is important to stress out and it consists in the format of the target labels.</p>
  <pre><code># Multi-class --&gt; one-hot encoded labels, only 1 label is correct   
    
  [1,0,0], [0,1,0], [0,0,1]
  
  # Multi-label --&gt; multiple labels can be correct
   
  [1,0,0], [1,1,0], [1,1,1], [0,1,0], [0,1,1], [0,0,1]
  </code></pre>
  <p>MSE is used when continuous values are predicted for some given inputs, therefore it belongs to the loss functions suitable for regression and it should not be used for your problem.</p>
  <p>Two loss functions that you could apply are <em><strong>Categorical Cross Entropy</strong></em> or <em><strong>Binary Cross Entropy</strong></em>. Despite being both based on cross-entropy, there is an important difference between them, consisting of the activation function they require.</p>
  <blockquote>
  <p><em><strong>Binary Cross Entropy</strong></em></p>
  </blockquote>
  <p><span class="math-container">$$L(y, \hat{y})=-\frac{1}{N} \sum_{i=0}^{N}\left(y * \log \left(\hat{y}_{i}\right)+(1-y) * \log \left(1-\hat{y}_{i}\right)\right)$$</span></p>
  <p>Despite the name that suggests this loss should be used only for binary classification, this is not strictly true and actually, this is the loss function that conceptually is best suited for multi-label tasks.</p>
  <p>Let's start with the binary classification case. We have a model that returns a single output score, to which the <em>sigmoid</em> function is applied in order to constrain the value between 0 and 1. Since we have a single score, the resulting value can be interpreted as a probability of belonging to one of the two classes, and the probability of being to the other class can be computed as 1 - value.</p>
  <p>What if we have multiple output scores, for example, 3 nodes for 3 classes? In this case, we still could apply the sigmoid function, ending up with three scores between 0 and 1.</p>
  <p><a href="https://i.sstatic.net/4Acyy.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/4Acyy.jpg" alt="enter image description here" /></a></p>
  <p>The important aspect to capture is that since the sigmoid treat each output node independently, the 3 scores would not sum up to 1, so they will represent 3 different probability distributions rather than a unique one. This means that each score after the sigmoid represents a distinct probability of belonging to that specific class. In the example above, for example, the prediction would be true for the two labels with a score higher than 0.5 and false for the remaining label. This also means that 3 different loss have to be computed, one for each possible output. In practice, what you'll be doing is to solve n binary classification problems where n is the number of possible labels.</p>
  <blockquote>
  <p><em><strong>Categorical Cross Entropy</strong></em></p>
  </blockquote>
  <p><span class="math-container">$$L(y, \hat{y})=-\sum_{j=0}^{M} \sum_{i=0}^{N}\left(y_{i j} * \log \left(\hat{y}_{i j}\right)\right)$$</span></p>
  <p>In categorical cross-entropy we apply the <em>softmax</em> function to the output scores of our model, to constrain them between 0 and 1 and to turn them into a probability distribution (they all sum to 1). The important thing to notice is that in this case, we end up with a unique distribution because, unlike the sigmoid function, the softmax consider all output scores together (they are summed in the denominator).</p>
  <p><a href="https://i.sstatic.net/Zbnvr.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Zbnvr.jpg" alt="enter image description here" /></a></p>
  <p>This implies that categorical cross-entropy is best suited for multi-class tasks, in which we end up with a single true prediction for each input instance. Nevertheless, this loss can also be applied also for multi-label tasks, as done in this <a href="https://research.fb.com/wp-content/uploads/2018/05/exploring_the_limits_of_weakly_supervised_pretraining.pdf?" rel="nofollow noreferrer">paper</a>. To do so, the authors turned each target vector into a uniform probability distribution, which means that the values of the true labels are not 1 but 1/k, with k being the total number of true labels.</p>
  <pre><code># Example of target vector tuned into uniform probability distribution
  [0, 1, 1] --&gt; [0, .5, .5]  
  [1, 1, 1] --&gt; [.33, .33, .33]
  </code></pre>
  <p>Note also that in the above-mentioned paper the authors found that categorical cross-entropy outperformed binary cross-entropy, even this is not a result that holds in general.</p>
  <p>Lastly, the are other loss that you could try which differ functions rather than cross-entropy, for example:</p>
  <blockquote>
  <p><em><strong>Hamming-Loss</strong></em></p>
  </blockquote>
  <p>It computes the fraction of wrong predicted labels</p>
  <p><span class="math-container">$$\frac{1}{|N| \cdot|L|} \sum_{i=1}^{|N|} \sum_{j=1}^{|L|} \operatorname{xor}\left(y_{i, j}, z_{i, j}\right)$$</span></p>
  <blockquote>
  <p><em><strong>Exact Match Ratio</strong></em></p>
  </blockquote>
  <p>Only predictions for which all target labels were correctly classified are considered correct.</p>
  <p><span class="math-container">$$ExactMatchRatio,\space M R=\frac{1}{n} \sum_{i=1}^{n} I\left(Y_{i}=Z_{i}\right)$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T14:55:42.090Z
Answers.Text: I think I may have been unclear. Each instance has only one desired subset. How is it a multi-label problem?
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T15:05:40.040Z
Answers.Text: @GiladDeutch So basically you're dealing with unique tuples of 6 elements (indices) for each input list? In this case the problem is simply multi-class classification and I would go for categorical cross-entropy. I would also change the representation of the target label in order to be hot-encoded. So if you have n possible tuples combination I would turn (1,0,0,0,0,0) into [1,0,..0] (length=n), then (1,1,0,0,0,0) into [0,1,0,..0] (length=n) and so on.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T19:05:55.470Z
Answers.Text: I'm not sure you follow. The 1's represent items from the list we want to choose: (1,1,0,0,0,0) would mean choosing the first two elements from the list, [0,1,0,..0] would mean choosing only the second, they're not the same.
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T19:21:26.440Z
Answers.Text: @GiladDeutch I do follow. I'll try to explain in a different way. First of all, you can use categorical cross entropy, it should work. Nevertheless I would **recode** the labels you currently have using **one hot encoding**. This means that you will transform each tuple containing 6 binary labels into a vector of size **n**, i.e. the total number of unique tuples in your dataset. So again, taking for example (1,1,0,0,0,0), it would be converted in a vector of length n containing a unique 1 value, such as [0,1,0,..n]. It would be just a mapping between tuples and one-hot encoded vectors.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T19:27:07.850Z
Answers.Text: The reason why I would do that is in the answer I wrote. Categorical cross entropy is meant to be used when only a single output node is suppose to be activated. If you use a tuple like (1,1,0,0,0,0) to calculate the error, you would train the model to maximise the output of the first 2 output nodes, which is ok but impractical. If you recode the tuples and train the model representing the tuples with one hot encoding instead, you would need **n** output nodes, and the model would learn to maximise the score of only one node, associated to a hot-encoding of one of the possible tuples.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T19:27:24.747Z
Answers.Text: if it is still not clear I can add a sketch to the answer.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T20:28:21.007Z
Answers.Text: Ok, I think I understand. But if I have subsets of a set of size 6 for example then the number of possible labels is the amount of possible subsets: n=2^6=64. So, one hot encoding would signifanctly enlarge the dimensions of the architecture. What you're saying is categorical cross entropy would work with the labels as they are, but it wouldn't be ideal?
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T20:43:57.250Z
Answers.Text: @GiladDeutch because as I explained in the answer, categorical cross entropy use the softmax, which turn all output scores into a unique prob distribution. This means that only one node is supposed to have the highest probability. Without one-hot labels there would be cases like (1,1,1,1,0,0) in which the model would be forced to predict a similar probability for 4 out of 6 nodes, so in the best case it will learn to predict .25 .25 .25 .25 0 0. which are not really high scores. With one hot encoding, you would allow the model to maximise a single output node per possible labels combination.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T20:45:53.640Z
Answers.Text: Nevertheless, if your dataset is not really large or you have other issue that prohibit one.hot encoding, you might try the approach used in the paper I linked. It would be like treating the problem as a multi-label task instead of a multi-class, but considering the format of your labels it will be conceptually the same, so I expect it to work.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T23:57:29.753Z
Answers.Text: I see your point, but it seems like using MSE loss and just taking each element with an output of more than 0.5 would much simpler. or if the subset size is known and is k, then just take the k elements with the highest outputs. Would that be a terrible solution in your opinion?
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T00:47:23.163Z
Answers.Text: @GiladDeutch The problems I see in using MSE is that the main purpose of it is to work with real values rather than binary ones. The MSE is a derivation of a linear regression, it assume the output variable to be a combination of the input values plus gaussian noise with constant mean and variance. If you use it in classification the residual will assume discrete values, in your case they would be 0, 1, 4, 9, 16, 25 or 36 and this why usually MSE is not used in classification. But of course in practice no one will stop you from trying!
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T10:59:06.453Z
Answers.Text: I see. What about the hamming and exact match ratio losses? I never heard of them but they seem like they could fit
Answers.UserDisplayName: Gilad Deutsch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-23T12:24:59.540Z
Answers.Text: @GiladDeutch if you use the hamming or EMR loss with your actually label format you'll be performing multi-label classification. The hamming loss for example would return an error of 2 for the two single vector [0,1,0,0] and [1,1,1,0]. If you treat them as classes the error will become 1, just because they are not equal. To be honest I do agree that one-hot encoding might be a pain in the ass and I think that treating the problem as multi-label is the simplest way to go. Even though you have unique combinations, this would be just an added helpful property of your dataset.
Answers.UserDisplayName: Edoardo Guerriero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-27T15:46:48.463Z
Answers.Text: Just a comment which is going to explain: I appreciate this answer; I had a kind of like question before and this totally cleared it up; Thanks!
Answers.UserDisplayName: William Martens
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-22T00:54:55.873Z
Answers.LastActivityDate: 2021-04-01T21:17:56.140Z
Answers.LastEditDate: 2021-04-01T21:17:56.140Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I'm trying to train a neural net to choose a subset from some list of objects. 
  The input is a list of objects <span class="math-container">$(a,b,c,d,e,f)$</span> and for each list of objects the label is a list composed of 0/1 - 1 for every object that is in the subset, for example <span class="math-container">$(1,1,0,1,0,1)$</span> represents choosing <span class="math-container">$a,b,d,f$</span>. I thought about using MSE loss to train the net but that seemed like a naive approach, is there some better loss function to use in this case?</p>
  
Tags: <neural-networks><deep-learning><objective-functions>
Title: Loss function for choosing a subset of objects
UserDisplayName: Gilad Deutsch
ThreadText: # Loss function for choosing a subset of objects
  
  (Asked by: username_0 on 2020-04-21)
  
  I'm trying to train a neural net to choose a subset from some list of objects.
  The input is a list of objects $(a,b,c,d,e,f)$ and for each list of objects the label is a list composed of 0/1 - 1 for every object that is in the subset, for example $(1,1,0,1,0,1)$ represents choosing $a,b,d,f$. I thought about using MSE loss to train the net but that seemed like a naive approach, is there some better loss function to use in this case?
  
  **Tags:** neural-networks, deep-learning, objective-functions
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-22. Score: 4
  
  The choice of the loss function depends primarily on the type of task you're tackling: classification or regression. Your problem is clearly a classification one since you have classes to which a given input can either belong or not. More specifically, what you're trying to do is multi-label classification, which is different from multi-class classification. The difference is important to stress out and it consists in the format of the target labels.
  
  ```
  # Multi-class --> one-hot encoded labels, only 1 label is correct   
    
  [1,0,0], [0,1,0], [0,0,1]
  
  # Multi-label --> multiple labels can be correct
   
  [1,0,0], [1,1,0], [1,1,1], [0,1,0], [0,1,1], [0,0,1]
  
  ```
  
  MSE is used when continuous values are predicted for some given inputs, therefore it belongs to the loss functions suitable for regression and it should not be used for your problem.
  
  Two loss functions that you could apply are ***Categorical Cross Entropy*** or ***Binary Cross Entropy***. Despite being both based on cross-entropy, there is an important difference between them, consisting of the activation function they require.
  
  > ***Binary Cross Entropy***
  
  $$L(y, \hat{y})=-\frac{1}{N} \sum\_{i=0}^{N}\left(y \* \log \left(\hat{y}\_{i}\right)+(1-y) \* \log \left(1-\hat{y}\_{i}\right)\right)$$
  
  Despite the name that suggests this loss should be used only for binary classification, this is not strictly true and actually, this is the loss function that conceptually is best suited for multi-label tasks.
  
  Let's start with the binary classification case. We have a model that returns a single output score, to which the *sigmoid* function is applied in order to constrain the value between 0 and 1. Since we have a single score, the resulting value can be interpreted as a probability of belonging to one of the two classes, and the probability of being to the other class can be computed as 1 - value.
  
  What if we have multiple output scores, for example, 3 nodes for 3 classes? In this case, we still could apply the sigmoid function, ending up with three scores between 0 and 1.
  
  [![enter image description here](https://i.sstatic.net/4Acyy.jpg)](https://i.sstatic.net/4Acyy.jpg)
  
  The important aspect to capture is that since the sigmoid treat each output node independently, the 3 scores would not sum up to 1, so they will represent 3 different probability distributions rather than a unique one. This means that each score after the sigmoid represents a distinct probability of belonging to that specific class. In the example above, for example, the prediction would be true for the two labels with a score higher than 0.5 and false for the remaining label. This also means that 3 different loss have to be computed, one for each possible output. In practice, what you'll be doing is to solve n binary classification problems where n is the number of possible labels.
  
  > ***Categorical Cross Entropy***
  
  $$L(y, \hat{y})=-\sum\_{j=0}^{M} \sum\_{i=0}^{N}\left(y\_{i j} \* \log \left(\hat{y}\_{i j}\right)\right)$$
  
  In categorical cross-entropy we apply the *softmax* function to the output scores of our model, to constrain them between 0 and 1 and to turn them into a probability distribution (they all sum to 1). The important thing to notice is that in this case, we end up with a unique distribution because, unlike the sigmoid function, the softmax consider all output scores together (they are summed in the denominator).
  
  [![enter image description here](https://i.sstatic.net/Zbnvr.jpg)](https://i.sstatic.net/Zbnvr.jpg)
  
  This implies that categorical cross-entropy is best suited for multi-class tasks, in which we end up with a single true prediction for each input instance. Nevertheless, this loss can also be applied also for multi-label tasks, as done in this [paper](https://research.fb.com/wp-content/uploads/2018/05/exploring_the_limits_of_weakly_supervised_pretraining.pdf?). To do so, the authors turned each target vector into a uniform probability distribution, which means that the values of the true labels are not 1 but 1/k, with k being the total number of true labels.
  
  ```
  # Example of target vector tuned into uniform probability distribution
  [0, 1, 1] --> [0, .5, .5]  
  [1, 1, 1] --> [.33, .33, .33]
  
  ```
  
  Note also that in the above-mentioned paper the authors found that categorical cross-entropy outperformed binary cross-entropy, even this is not a result that holds in general.
  
  Lastly, the are other loss that you could try which differ functions rather than cross-entropy, for example:
  
  > ***Hamming-Loss***
  
  It computes the fraction of wrong predicted labels
  
  $$\frac{1}{|N| \cdot|L|} \sum\_{i=1}^{|N|} \sum\_{j=1}^{|L|} \operatorname{xor}\left(y\_{i, j}, z\_{i, j}\right)$$
  
  > ***Exact Match Ratio***
  
  Only predictions for which all target labels were correctly classified are considered correct.
  
  $$ExactMatchRatio,\space M R=\frac{1}{n} \sum\_{i=1}^{n} I\left(Y\_{i}=Z\_{i}\right)$$
  
  > **Comments:**
  > **username_0**: I think I may have been unclear. Each instance has only one desired subset. How is it a multi-label problem?  
  > **username_1**: @GiladDeutch So basically you're dealing with unique tuples of 6 elements (indices) for each input list? In this case the problem is simply multi-class classification and I would go for categorical cross-entropy. I would also change the representation of the target label in order to be hot-encoded. So if you have n possible tuples combination I would turn (1,0,0,0,0,0) into [1,0,..0] (length=n), then (1,1,0,0,0,0) into [0,1,0,..0] (length=n) and so on.  
  > **username_0**: I'm not sure you follow. The 1's represent items from the list we want to choose: (1,1,0,0,0,0) would mean choosing the first two elements from the list, [0,1,0,..0] would mean choosing only the second, they're not the same.  
  > **username_1**: @GiladDeutch I do follow. I'll try to explain in a different way. First of all, you can use categorical cross entropy, it should work. Nevertheless I would **recode** the labels you currently have using **one hot encoding**. This means that you will transform each tuple containing 6 binary labels into a vector of size **n**, i.e. the total number of unique tuples in your dataset. So again, taking for example (1,1,0,0,0,0), it would be converted in a vector of length n containing a unique 1 value, such as [0,1,0,..n]. It would be just a mapping between tuples and one-hot encoded vectors.  
  > **username_1**: The reason why I would do that is in the answer I wrote. Categorical cross entropy is meant to be used when only a single output node is suppose to be activated. If you use a tuple like (1,1,0,0,0,0) to calculate the error, you would train the model to maximise the output of the first 2 output nodes, which is ok but impractical. If you recode the tuples and train the model representing the tuples with one hot encoding instead, you would need **n** output nodes, and the model would learn to maximise the score of only one node, associated to a hot-encoding of one of the possible tuples.  
  > **username_1**: if it is still not clear I can add a sketch to the answer.  
  > **username_0**: Ok, I think I understand. But if I have subsets of a set of size 6 for example then the number of possible labels is the amount of possible subsets: n=2^6=64. So, one hot encoding would signifanctly enlarge the dimensions of the architecture. What you're saying is categorical cross entropy would work with the labels as they are, but it wouldn't be ideal?  
  > **username_1**: @GiladDeutch because as I explained in the answer, categorical cross entropy use the softmax, which turn all output scores into a unique prob distribution. This means that only one node is supposed to have the highest probability. Without one-hot labels there would be cases like (1,1,1,1,0,0) in which the model would be forced to predict a similar probability for 4 out of 6 nodes, so in the best case it will learn to predict .25 .25 .25 .25 0 0. which are not really high scores. With one hot encoding, you would allow the model to maximise a single output node per possible labels combination.  
  > **username_1**: Nevertheless, if your dataset is not really large or you have other issue that prohibit one.hot encoding, you might try the approach used in the paper I linked. It would be like treating the problem as a multi-label task instead of a multi-class, but considering the format of your labels it will be conceptually the same, so I expect it to work.  
  > **username_0**: I see your point, but it seems like using MSE loss and just taking each element with an output of more than 0.5 would much simpler. or if the subset size is known and is k, then just take the k elements with the highest outputs. Would that be a terrible solution in your opinion?  
  > **username_1**: @GiladDeutch The problems I see in using MSE is that the main purpose of it is to work with real values rather than binary ones. The MSE is a derivation of a linear regression, it assume the output variable to be a combination of the input values plus gaussian noise with constant mean and variance. If you use it in classification the residual will assume discrete values, in your case they would be 0, 1, 4, 9, 16, 25 or 36 and this why usually MSE is not used in classification. But of course in practice no one will stop you from trying!  
  > **username_0**: I see. What about the hamming and exact match ratio losses? I never heard of them but they seem like they could fit  
  > **username_1**: @GiladDeutch if you use the hamming or EMR loss with your actually label format you'll be performing multi-label classification. The hamming loss for example would return an error of 2 for the two single vector [0,1,0,0] and [1,1,1,0]. If you treat them as classes the error will become 1, just because they are not equal. To be honest I do agree that one-hot encoding might be a pain in the ass and I think that treating the problem as multi-label is the simplest way to go. Even though you have unique combinations, this would be just an added helpful property of your dataset.  
  > **username_2**: Just a comment which is going to explain: I appreciate this answer; I had a kind of like question before and this totally cleared it up; Thanks!  
  
  ---
  
  