Answers.Body: <p>This got answered at <a href="https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290" rel="nofollow noreferrer">https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290</a>, derivation for (2) formula are available at <a href="https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136" rel="nofollow noreferrer">https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-28T18:05:09.747Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-28T07:03:40.597Z
Answers.LastActivityDate: 2022-10-28T07:03:40.597Z
Answers.UserDisplayName: src091
Body: <p><a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java" rel="nofollow noreferrer">SigmoidBinaryCrossEntropyLoss</a> implementation in DJL accepts two kinds of outputs from NNs:</p>
  <ol>
  <li>where <code>sigmoid</code> activation has already been applied.</li>
  <li>where raw NN output is taken as is.</li>
  </ol>
  <p>The choice is determied by <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L50" rel="nofollow noreferrer">fromSigmoid</a> parameter.</p>
  <p>Here's how (1) <code>fromSigmoid=true</code> <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L72-L75" rel="nofollow noreferrer">looks like</a>:</p>
  <pre><code>loss =
        epsLog(pred)
        .mul(lab)
        .add(epsLog(NDArrays.sub(1., pred)).mul(NDArrays.sub(1., lab)));
  </code></pre>
  <p>And here's how (2) <code>fromSigmoid=false</code> <a href="https://github.com/deepjavalibrary/djl/blob/master/api/src/main/java/ai/djl/training/loss/SigmoidBinaryCrossEntropyLoss.java#L67-L70" rel="nofollow noreferrer">looks like</a>:</p>
  <pre><code>loss =
        Activation.relu(pred)
        .sub(pred.mul(lab))
        .add(Activation.softPlus(pred.abs().neg()));
  </code></pre>
  <p>The logic behind (1) <code>fromSigmoid=true</code> branch looks like a standard cross entropy loss implementation, but I fail to understand why (2) <code>fromSigmoid=false</code> is implemented the way it is.</p>
  <p>For instance, why is there no <code>sigmoid</code> application, why is there <code>relu</code> later followed by <code>softPlus</code>? I'd like to understand what does (2) do and what's the theory behind its implementation.</p>
  
Tags: <objective-functions><loss><sigmoid><binary-crossentropy>
Title: Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
UserDisplayName: src091
ThreadText: # Why is `SigmoidBinaryCrossEntropyLoss` in `DJL` implemented this way?
  
  (Asked by: username_0 on 2022-10-19)
  
  SigmoidBinaryCrossEntropyLoss implementation in DJL accepts two kinds of outputs from NNs:
  
  where sigmoid activation has already been applied.
  where raw NN output is taken as is.
  
  The choice is determied by fromSigmoid parameter.
  Here's how (1) fromSigmoid=true looks like:
  loss =
        epsLog(pred)
        .mul(lab)
        .add(epsLog(NDArrays.sub(1., pred)).mul(NDArrays.sub(1., lab)));
  
  And here's how (2) fromSigmoid=false looks like:
  loss =
        Activation.relu(pred)
        .sub(pred.mul(lab))
        .add(Activation.softPlus(pred.abs().neg()));
  
  The logic behind (1) fromSigmoid=true branch looks like a standard cross entropy loss implementation, but I fail to understand why (2) fromSigmoid=false is implemented the way it is.
  For instance, why is there no sigmoid application, why is there relu later followed by softPlus? I'd like to understand what does (2) do and what's the theory behind its implementation.
  
  **Tags:** objective-functions, loss, sigmoid, binary-crossentropy
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-10-28. Score: 0 | Accepted answer
  
  This got answered at <https://github.com/deepjavalibrary/djl/discussions/2087#discussioncomment-3955290>, derivation for (2) formula are available at <https://github.com/tensorflow/tensorflow/blob/r1.1/tensorflow/python/ops/nn_impl.py#L100-L136>.
  
  > **Comments:**
  > **username_1**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  