Answers.Body: <p>This is multi-label classification, which means you have two binary classification problems, one for each of your classes. This is different than multi-class classification.</p>
  <p>For this use binary cross-entropy loss, with binary encoded labels (0/1), not one-hot encoded labels, and put a sigmoid activation at the output with two neurons (one for each class).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T12:19:59.683Z
Answers.Text: Thank you. To make this work optimally, is there any issue that both classes occur very scarcely in the training set? I.e. very unbalanced? Would something like FocalLoss be a solution? I can create a third 'normal' class as class 1 and 2 are mutually exclusive. Trying to find what would perform best.
Answers.UserDisplayName: dorien
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T02:59:25.817Z
Answers.LastActivityDate: 2023-03-19T02:59:25.817Z
Answers.UserDisplayName: Dr. Snoopy
Body: <p>I have a neural network that predicts 2 classes of a time series (bottom and top). Currenlty my Y labels are size 2: [1 0] for bottom and [0 1] for top. The NN has 2 output nodes.</p>
  <p>Of course not every instance is a bottom or top. (Overwhelmingly) most of the time we have no bottom and no top.</p>
  <p>Should I change my network (and Y labels) to have 3 outputs?</p>
  <p>And what would be appropriate as a loss function. I guess with 3 outputs I could use entropy or focal loss. With 2 outputs I don't think entropy is correct.</p>
  <p>Currently I use PyTorch (lightening), and I have:</p>
  <pre><code>self.lstm = nn.LSTM(input_size=n_features, 
                              hidden_size=hidden_size,
                              num_layers=num_layers, 
                              dropout=dropout, 
                              batch_first=True)
  self.linear = nn.Linear(hidden_size, 2)
          
  
  
  def forward(self, x):
          lstm_out, _ = self.lstm(x)
          y_pred = self.linear(lstm_out[:,-1])
          return y_pred
  
  criterion = nn.CrossEntropyLoss()
  </code></pre>
  
Tags: <neural-networks><classification><objective-functions><pytorch>
Title: Which loss / activation function with 2 classes that do not occur often and do not sum to one?
UserDisplayName: dorien
ThreadText: # Which loss / activation function with 2 classes that do not occur often and do not sum to one?
  
  (Asked by: username_0 on 2023-03-18)
  
  I have a neural network that predicts 2 classes of a time series (bottom and top). Currenlty my Y labels are size 2: [1 0] for bottom and [0 1] for top. The NN has 2 output nodes.
  Of course not every instance is a bottom or top. (Overwhelmingly) most of the time we have no bottom and no top.
  Should I change my network (and Y labels) to have 3 outputs?
  And what would be appropriate as a loss function. I guess with 3 outputs I could use entropy or focal loss. With 2 outputs I don't think entropy is correct.
  Currently I use PyTorch (lightening), and I have:
  self.lstm = nn.LSTM(input_size=n_features, 
                              hidden_size=hidden_size,
                              num_layers=num_layers, 
                              dropout=dropout, 
                              batch_first=True)
  self.linear = nn.Linear(hidden_size, 2)
          
  
  
  def forward(self, x):
          lstm_out, _ = self.lstm(x)
          y_pred = self.linear(lstm_out[:,-1])
          return y_pred
  
  criterion = nn.CrossEntropyLoss()
  
  **Tags:** neural-networks, classification, objective-functions, pytorch
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-19. Score: 1 | Accepted answer
  
  This is multi-label classification, which means you have two binary classification problems, one for each of your classes. This is different than multi-class classification.
  
  For this use binary cross-entropy loss, with binary encoded labels (0/1), not one-hot encoded labels, and put a sigmoid activation at the output with two neurons (one for each class).
  
  > **Comments:**
  > **username_0**: Thank you. To make this work optimally, is there any issue that both classes occur very scarcely in the training set? I.e. very unbalanced? Would something like FocalLoss be a solution? I can create a third 'normal' class as class 1 and 2 are mutually exclusive. Trying to find what would perform best.  
  
  ---
  
  