Answers.Body: <p>This got answered at <a href="https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290" rel="nofollow noreferrer">https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290</a>, derivation for (2) formula are available at <a href="https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-28T18:05:09.747Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-28T07:03:40.597Z
Answers.LastActivityDate: 2022-10-28T07:03:40.597Z
Answers.UserDisplayName: src091
Body: <p><a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java" rel="nofollow noreferrer">SigmoidBinaryCrossEntropyLoss</a> implementation in DJL accepts two kinds of outputs from NNs:</p>
  <ol>
  <li>where <code>sigmoid</code> activation has already been applied.</li>
  <li>where raw NN output is taken as is.</li>
  </ol>
  <p>The choice is determied by <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L50" rel="nofollow noreferrer">fromSigmoid</a> parameter.</p>
  <p>Here's how (1) <code>fromSigmoid=true</code> <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L72-L75" rel="nofollow noreferrer">looks like</a>:</p>
  <pre><code>loss =
        epsLog(pred)
        .mul(lab)
        .add(epsLog(NDArrays.sub(1., pred)).mul(NDArrays.sub(1., lab)));
  </code></pre>
  <p>And here's how (2) <code>fromSigmoid=false</code> <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L67-L70" rel="nofollow noreferrer">looks like</a>:</p>
  <pre><code>loss =
        Activation.relu(pred)
        .sub(pred.mul(lab))
        .add(Activation.softPlus(pred.abs().neg()));
  </code></pre>
  <p>The logic behind (1) <code>fromSigmoid=true</code> branch looks like a standard cross entropy loss implementation, but I fail to understand why (2) <code>fromSigmoid=false</code> is implemented the way it is.</p>
  <p>For instance, why is there no <code>sigmoid</code> application, why is there <code>relu</code> later followed by <code>softPlus</code>? I'd like to understand what does (2) do and what's the theory behind its implementation.</p>
  
Tags: <objective-functions><loss><sigmoid><binary-crossentropy>
Title: Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
UserDisplayName: src091
ThreadText: # Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
  
  (Asked by: username_0 on 2022-10-19)
  
  [SigmoidBinaryCrossEntropyLoss](https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java) implementation in DJL accepts two kinds of outputs from NNs:
  
  1. where `sigmoid` activation has already been applied.
  2. where raw NN output is taken as is.
  
  The choice is determied by [fromSigmoid](https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L50) parameter.
  
  Here's how (1) `fromSigmoid=true` [looks like](https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L72-L75):
  
  ```
  loss =
        epsLog(pred)
        .mul(lab)
        .add(epsLog(NDArrays.sub(1., pred)).mul(NDArrays.sub(1., lab)));
  
  ```
  
  And here's how (2) `fromSigmoid=false` [looks like](https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L67-L70):
  
  ```
  loss =
        Activation.relu(pred)
        .sub(pred.mul(lab))
        .add(Activation.softPlus(pred.abs().neg()));
  
  ```
  
  The logic behind (1) `fromSigmoid=true` branch looks like a standard cross entropy loss implementation, but I fail to understand why (2) `fromSigmoid=false` is implemented the way it is.
  
  For instance, why is there no `sigmoid` application, why is there `relu` later followed by `softPlus`? I'd like to understand what does (2) do and what's the theory behind its implementation.
  
  **Tags:** objective-functions, loss, sigmoid, binary-crossentropy
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-10-28. Score: 0 | Accepted answer
  
  This got answered at <https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290>, derivation for (2) formula are available at <https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136>.
  
  > **Comments:**
  > **username_1**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  