Answers.Body: <p>I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the article is better and less prone to errors than using the full derivation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-02T10:27:22.283Z
Answers.LastActivityDate: 2022-06-02T10:27:22.283Z
Answers.UserDisplayName: user605734 MBS
Body: <p>so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using softmax and crossentropy instead.</p>
  <p>I did the derivation of the formulas as in <a href="https://towardsdatascience.com/derivative-of-the-softmax-function-and-the-categorical-cross-entropy-loss-ffceefc081d1" rel="nofollow noreferrer">this article</a>, but I tried computing the derivative of the cost function w.r.t. its inputs and the Jacobian of the softmax outputs w.r.t. the inputs separately.</p>
  <p>This is what I did:</p>
  <p>So, let <span class="math-container">$Y$</span> be the matrix with rows equal to the expected class probabilities, <span class="math-container">$\hat{Y}$</span> the outputs of the softmax layer, <span class="math-container">$X$</span> the inputs of the softmax, all of these matrices of size <span class="math-container">$n\times c$</span> (samples times classes). The cost function <span class="math-container">$J(Y,\hat{Y}) = -sum(Y * \log(\hat{Y}))$</span> where * is element by element multiplication.</p>
  <p>Then, if <span class="math-container">$E$</span> is the cost function, I first compute</p>
  <p><span class="math-container">$$\frac{\partial E}{\partial \hat{Y}_j^{(i)}} = -\frac{Y_j^{(i)}}{\hat{Y}_j^{(i)}},\:\: i=1,\cdots,n : j=1,\cdots,c$$</span></p>
  <p>This is a <span class="math-container">$n\times c$</span> matrix. Then, being <span class="math-container">$S$</span> the softMax function I compute the Jacobian of the softmax at each row of <span class="math-container">$X$</span>, so a total of <span class="math-container">$n$</span> matrices of <span class="math-container">$c\times c$</span> dimensions.</p>
  <p><span class="math-container">$$JS^{(i)} = \left(\frac{\partial \hat{Y}_j^{(i)}}{\partial X_k^{(i)}} \right)_{j,k=1,\cdots,c},\:\: i=1,\cdots,n$$</span></p>
  <p>Finally to compute the partial derivative of the error with respect to the inputs of the softmax I multiply the rows of <span class="math-container">$\frac{\partial E}{\partial \hat{Y}}$</span> with each jacobian, to get a new <span class="math-container">$n\times c$</span> matrix.</p>
  <p>This way I have everything needed to compute:</p>
  <p><span class="math-container">$$\frac{\partial E}{\partial X_j^{(i)}}= \sum_{k=1}^c \frac{\partial E}{\partial \hat{Y}_k^{(i)}}\frac{\partial \hat{Y}_k^{(i)}}{\partial X_j^{(i)}} ,\:\: i=1,\cdots,n : j=1,\cdots,c$$</span></p>
  <p>Is this correct? Where is the problem if not so? Might the numerical errors make everything fail?</p>
  <p>Thanks in advance.</p>
  
Tags: <backpropagation><cross-entropy><softmax>
Title: Trouble writing the backpropagation algorithm in python through crossentropy and softmax
UserDisplayName: user605734 MBS
ThreadText: # Trouble writing the backpropagation algorithm in python through crossentropy and softmax
  
  (Asked by: username_0 on 2022-06-01)
  
  so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using softmax and crossentropy instead.
  
  I did the derivation of the formulas as in [this article](https://towardsdatascience.com/derivative-of-the-softmax-function-and-the-categorical-cross-entropy-loss-ffceefc081d1), but I tried computing the derivative of the cost function w.r.t. its inputs and the Jacobian of the softmax outputs w.r.t. the inputs separately.
  
  This is what I did:
  
  So, let $Y$ be the matrix with rows equal to the expected class probabilities, $\hat{Y}$ the outputs of the softmax layer, $X$ the inputs of the softmax, all of these matrices of size $n\times c$ (samples times classes). The cost function $J(Y,\hat{Y}) = -sum(Y \* \log(\hat{Y}))$ where \* is element by element multiplication.
  
  Then, if $E$ is the cost function, I first compute
  
  $$\frac{\partial E}{\partial \hat{Y}\_j^{(i)}} = -\frac{Y\_j^{(i)}}{\hat{Y}\_j^{(i)}},\:\: i=1,\cdots,n : j=1,\cdots,c$$
  
  This is a $n\times c$ matrix. Then, being $S$ the softMax function I compute the Jacobian of the softmax at each row of $X$, so a total of $n$ matrices of $c\times c$ dimensions.
  
  $$JS^{(i)} = \left(\frac{\partial \hat{Y}\_j^{(i)}}{\partial X\_k^{(i)}} \right)\_{j,k=1,\cdots,c},\:\: i=1,\cdots,n$$
  
  Finally to compute the partial derivative of the error with respect to the inputs of the softmax I multiply the rows of $\frac{\partial E}{\partial \hat{Y}}$ with each jacobian, to get a new $n\times c$ matrix.
  
  This way I have everything needed to compute:
  
  $$\frac{\partial E}{\partial X\_j^{(i)}}= \sum\_{k=1}^c \frac{\partial E}{\partial \hat{Y}\_k^{(i)}}\frac{\partial \hat{Y}\_k^{(i)}}{\partial X\_j^{(i)}} ,\:\: i=1,\cdots,n : j=1,\cdots,c$$
  
  Is this correct? Where is the problem if not so? Might the numerical errors make everything fail?
  
  Thanks in advance.
  
  **Tags:** backpropagation, cross-entropy, softmax
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2022-06-02. Score: 1 | Accepted answer
  
  I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the article is better and less prone to errors than using the full derivation.
  
  
  ---
  
  