Answers.Body: <p>Your call to <code>model.predict()</code> is returning the <em>logits</em> for softmax. This is useful for training purposes.</p>
  <p>To get probabilties, you need to apply softmax on the logits.</p>
  <pre><code>import torch.nn.functional as F
  logits = model.predict()
  probabilities = F.softmax(logits, dim=-1)
  </code></pre>
  <p>Now you can apply your threshold same as for the Keras model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-28T18:01:35.900Z
Answers.Text: Can you elaborate on how logits are used for training? All of the models I've used apply loss to the output of the softmax layer, I've never seen anything that uses the raw logits for calculating loss
Answers.UserDisplayName: A Tyshka
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-28T20:25:55.387Z
Answers.Text: @ATyshka: Both Tensorflow and Pytorch provide this feature as standard, and I expect most other NN libraries do as well. As the original question was about Pytorch, here's a link for the Pytorch loss function for binary cross entropy using the raw logits: https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html  - the OP already was using this or similar, so I don't see any need to put that in the answer. Hope it helps though - if you are running loss on the ouput layer, it is less efficient (and possibly unstable), so switch if you can
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-03T08:12:31.063Z
Answers.LastActivityDate: 2021-03-03T08:12:31.063Z
Answers.UserDisplayName: Neil Slater
Body: <p>I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.</p>
  <p>I have a classification task. The training data has 50 different labels. The customer wants to differentiate the low probability predictions, meaning that, I have to classify some test data as &quot;Unclassified / Other&quot; depending on the probability (certainty?) of the model.</p>
  <p>When I test my code, the prediction result is a numpy array. One example is:</p>
  <pre><code>[[-1.7862008  -0.7037363   0.09885322  1.5318055   2.1137428  -0.2216074
     0.18905772 -0.32575375  1.0748093  -0.06001111  0.01083148  0.47495762
     0.27160102  0.13852511 -0.68440574  0.6773654  -2.2712054  -0.2864312
    -0.8428862  -2.1132915  -1.0157436  -1.0340284  -0.35126117 -1.0333195
     9.149789   -0.21288703  0.11455813 -0.32903734  0.10503325 -0.3004114
    -1.3854568  -0.01692022 -0.4388664  -0.42163098 -0.09182278 -0.28269592
    -0.33082992 -1.147654   -0.6703184   0.33038092 -0.50087476  1.1643585
     0.96983343  1.3400391   1.0692116  -0.7623776  -0.6083422  -0.91371405
     0.10002492]]
  </code></pre>
  <p>I'm then using <code>numpy.argmax()</code> to identify the correct label.</p>
  <p>My question is, is it possible to define a threshold (say, 0.6), and then compare the probability of the argmax() element so that I can classify the prediction as &quot;other&quot; if the probability is less than the threshold value?</p>
  <hr />
  <p><strong>Edit 1:</strong></p>
  <p>We are using 2 different models. One is Keras, and the other is BertTransformer. We have no problem in Keras since it gives the probabilities so I'm skipping Keras model.</p>
  <p>The Bert model is pretrained. Here is how it is generated:</p>
  <pre><code>def model(self, data):
          number_of_categories = len(data['encoded_categories'].unique())
          model = BertForSequenceClassification.from_pretrained(
              &quot;dbmdz/bert-base-turkish-128k-uncased&quot;,
              num_labels=number_of_categories,
              output_attentions=False,
              output_hidden_states=False,
          )
  
          # model.cuda()
  
          return model
  </code></pre>
  <p>The output given above is the result of <code>model.predict()</code> method. We compare both models, Bert is slightly ahead, therefore we know that the prediction works just fine. However, we are not sure what those numbers signify or represent.</p>
  <p>Here is the <a href="https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification" rel="nofollow noreferrer">Bert documentation</a>.</p>
  
Tags: <classification><python><bert><probability><multiclass-classification>
Title: How do I calculate the probabilities of the BERT model prediction logits?
UserDisplayName: iso_9001_
ThreadText: # How do I calculate the probabilities of the BERT model prediction logits?
  
  (Asked by: username_0 on 2021-03-02)
  
  I might be getting this completely wrong, but please let me first try to explain what I need, and then what's wrong.
  I have a classification task. The training data has 50 different labels. The customer wants to differentiate the low probability predictions, meaning that, I have to classify some test data as "Unclassified / Other" depending on the probability (certainty?) of the model.
  When I test my code, the prediction result is a numpy array. One example is:
  [[-1.7862008  -0.7037363   0.09885322  1.5318055   2.1137428  -0.2216074
     0.18905772 -0.32575375  1.0748093  -0.06001111  0.01083148  0.47495762
     0.27160102  0.13852511 -0.68440574  0.6773654  -2.2712054  -0.2864312
    -0.8428862  -2.1132915  -1.0157436  -1.0340284  -0.35126117 -1.0333195
     9.149789   -0.21288703  0.11455813 -0.32903734  0.10503325 -0.3004114
    -1.3854568  -0.01692022 -0.4388664  -0.42163098 -0.09182278 -0.28269592
    -0.33082992 -1.147654   -0.6703184   0.33038092 -0.50087476  1.1643585
     0.96983343  1.3400391   1.0692116  -0.7623776  -0.6083422  -0.91371405
     0.10002492]]
  
  I'm then using numpy.argmax() to identify the correct label.
  My question is, is it possible to define a threshold (say, 0.6), and then compare the probability of the argmax() element so that I can classify the prediction as "other" if the probability is less than the threshold value?
  
  Edit 1:
  We are using 2 different models. One is Keras, and the other is BertTransformer. We have no problem in Keras since it gives the probabilities so I'm skipping Keras model.
  The Bert model is pretrained. Here is how it is generated:
  def model(self, data):
          number_of_categories = len(data['encoded_categories'].unique())
          model = BertForSequenceClassification.from_pretrained(
              "dbmdz/bert-base-turkish-128k-uncased",
              num_labels=number_of_categories,
              output_attentions=False,
              output_hidden_states=False,
          )
  
          # model.cuda()
  
          return model
  
  The output given above is the result of model.predict() method. We compare both models, Bert is slightly ahead, therefore we know that the prediction works just fine. However, we are not sure what those numbers signify or represent.
  Here is the Bert documentation.
  
  **Tags:** classification, python, bert, probability, multiclass-classification
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-03. Score: 3 | Accepted answer
  
  Your call to model.predict() is returning the logits for softmax. This is useful for training purposes.
  To get probabilties, you need to apply softmax on the logits.
  import torch.nn.functional as F
  logits = model.predict()
  probabilities = F.softmax(logits, dim=-1)
  
  Now you can apply your threshold same as for the Keras model.
  
  > **Comments:**
  > **username_2**: Can you elaborate on how logits are used for training? All of the models I've used apply loss to the output of the softmax layer, I've never seen anything that uses the raw logits for calculating loss  
  > **username_1**: @ATyshka: Both Tensorflow and Pytorch provide this feature as standard, and I expect most other NN libraries do as well. As the original question was about Pytorch, here's a link for the Pytorch loss function for binary cross entropy using the raw logits: https://pytorch.org/docs/stable/generated/torch.nn.BCEWithLogitsLoss.html  - the OP already was using this or similar, so I don't see any need to put that in the answer. Hope it helps though - if you are running loss on the ouput layer, it is less efficient (and possibly unstable), so switch if you can  
  
  ---
  
  