Answers.Body: <p>Alright. Consider an ordinary neural network, so, in the last layer, we have, <span class="math-container">$z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$</span>, where <span class="math-container">$a^{[L]} = \sigma(z^{[L]})$</span>, where <span class="math-container">$\sigma$</span> is the softmax activation:
  <span class="math-container">$$
  \sigma(\mathbf z)_{i} = \frac{e^{z_i}}{\sum_k e^{z_k}}
  $$</span></p>
  <p>I think, one of the most effective ways of not to get confused about all these matrices with different dimensions, is to simply get rid of the matrices and do all calculations componentwise. So, calculating the <span class="math-container">$ij$</span> component of the jacobian matrix, we have:
  <span class="math-container">$$
  J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{e^{z_i}}{\sum_k e^{z_k}}\delta_{ij} - \frac{e^{z_i}e^{z_j}}{(\sum_k e^{z_k})^2}
  $$</span></p>
  <p>Consider a cost <span class="math-container">$C$</span>, and we wish to calculate the derivative with respect to the weights of the last layer, that is, we are running the first step of backpropagation. Applying chain rule to the <span class="math-container">$ij$</span>-component of the weight:</p>
  <p><span class="math-container">$$
  \frac{\partial C}{\partial W^{[L]}_{ij}} 
  = \sum_k\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}}
  $$</span></p>
  <p>And, with one more chain rule:
  <span class="math-container">$$
  \frac{\partial a^{[L]}_{k}}{\partial W^{[L]}_{ij}} = 
  \sum_s\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
  $$</span></p>
  <p>Now we can identify exactly where the jacobian matrix is:
  <span class="math-container">$$
  J_{ij} = \frac{\partial\sigma_i}{\partial z_j} = \frac{\partial a^{[L]}_{i}}{\partial z^{[L]}_j}
  $$</span></p>
  <p>Thus, the back-propagation step is just:
  <span class="math-container">$$
  \frac{\partial C}{\partial W^{[L]}_{ij}} 
  = \sum_k\sum_s\frac{\partial C}{\partial a_{k}}\frac{\partial a^{[L]}_{k}}{\partial z^{[L]}_s}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
  = \sum_k\sum_s\frac{\partial C}{\partial a_{k}}J_{ks}\frac{\partial z^{[L]}_s}{\partial W^{[L]}_{ij}}
  $$</span></p>
  <p>Now, in component language, it is much simpler to identify where are all the matrix multiplications: The transpose of the gradient with respect to the cost function, multiplied by the jacobian matrix of the softmax activation (or whatever activation), and finally, the last derivative, which will evaluate to something depending on the activation of the previous layer.</p>
  <p>=].</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-21T05:06:37.260Z
Answers.LastActivityDate: 2021-01-22T00:06:00.443Z
Answers.LastEditDate: 2021-01-22T00:06:00.443Z
Answers.UserDisplayName: kid
Body: <p>I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-class classifications.</p>
  <p>What confuses me mainly are the matrix-vector shapes and their multiplications, and their implementations in code.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T11:26:22.790Z
Comments.Text: There's no shortage of tutorials on the matter. You need to be more specific about what gaps they didn't bridge
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-16T19:06:47.427Z
Comments.Text: @Alex what confuses me mainly is how the matrix shapes are matched, during backpropagation I saw the derivative of softmax being multiplied element wise with the gradient of the loss function. However since softmax's gradient is a Jacobian - a 2D matrix, I am confused as to how this is multiplied with a 1D vector
Comments.UserDisplayName: Ilknur Mustafa
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-27T00:24:42.793Z
Comments.Text: I don't know if this will really be useful, but maybe take a look at [Notes on Backpropagation](https://www.ics.uci.edu/~pjsadows/notes.pdf) by Peter Sadowski.
Comments.UserDisplayName: nbro
Tags: <deep-learning><backpropagation><math><cross-entropy><softmax>
Title: How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
UserDisplayName: Ilknur Mustafa
ThreadText: # How to compute the gradient of the cross-entropy loss function with respect to the parameters with softmax activation function?
  
  (Asked by: username_0 on 2021-01-16)
  
  I've seen plenty of examples of people doing Sigmoid + MSE backpropagation implementations, yet I do not seem to understand how to implement backpropagation as stated in the title in the case of multi-class classifications.
  
  What confuses me mainly are the matrix-vector shapes and their multiplications, and their implementations in code.
  
  **Tags:** deep-learning, backpropagation, math, cross-entropy, softmax
  > **Comments:**
  > **username_1**: There's no shortage of tutorials on the matter. You need to be more specific about what gaps they didn't bridge  
  > **username_0**: @Alex what confuses me mainly is how the matrix shapes are matched, during backpropagation I saw the derivative of softmax being multiplied element wise with the gradient of the loss function. However since softmax's gradient is a Jacobian - a 2D matrix, I am confused as to how this is multiplied with a 1D vector  
  > **username_2**: I don't know if this will really be useful, but maybe take a look at [Notes on Backpropagation](https://www.ics.uci.edu/~pjsadows/notes.pdf) by Peter Sadowski.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-01-21. Score: 1
  
  Alright. Consider an ordinary neural network, so, in the last layer, we have, $z^{[L]} = W^{[L]} a^{[L-1]} + b^{[L]}$, where $a^{[L]} = \sigma(z^{[L]})$, where $\sigma$ is the softmax activation:
  $$
  \sigma(\mathbf z)\_{i} = \frac{e^{z\_i}}{\sum\_k e^{z\_k}}
  $$
  
  I think, one of the most effective ways of not to get confused about all these matrices with different dimensions, is to simply get rid of the matrices and do all calculations componentwise. So, calculating the $ij$ component of the jacobian matrix, we have:
  $$
  J\_{ij} = \frac{\partial\sigma\_i}{\partial z\_j} = \frac{e^{z\_i}}{\sum\_k e^{z\_k}}\delta\_{ij} - \frac{e^{z\_i}e^{z\_j}}{(\sum\_k e^{z\_k})^2}
  $$
  
  Consider a cost $C$, and we wish to calculate the derivative with respect to the weights of the last layer, that is, we are running the first step of backpropagation. Applying chain rule to the $ij$-component of the weight:
  
  $$
  \frac{\partial C}{\partial W^{[L]}\_{ij}}
  = \sum\_k\frac{\partial C}{\partial a\_{k}}\frac{\partial a^{[L]}\_{k}}{\partial W^{[L]}\_{ij}}
  $$
  
  And, with one more chain rule:
  $$
  \frac{\partial a^{[L]}\_{k}}{\partial W^{[L]}\_{ij}} =
  \sum\_s\frac{\partial a^{[L]}\_{k}}{\partial z^{[L]}\_s}\frac{\partial z^{[L]}\_s}{\partial W^{[L]}\_{ij}}
  $$
  
  Now we can identify exactly where the jacobian matrix is:
  $$
  J\_{ij} = \frac{\partial\sigma\_i}{\partial z\_j} = \frac{\partial a^{[L]}\_{i}}{\partial z^{[L]}\_j}
  $$
  
  Thus, the back-propagation step is just:
  $$
  \frac{\partial C}{\partial W^{[L]}\_{ij}}
  = \sum\_k\sum\_s\frac{\partial C}{\partial a\_{k}}\frac{\partial a^{[L]}\_{k}}{\partial z^{[L]}\_s}\frac{\partial z^{[L]}\_s}{\partial W^{[L]}\_{ij}}
  = \sum\_k\sum\_s\frac{\partial C}{\partial a\_{k}}J\_{ks}\frac{\partial z^{[L]}\_s}{\partial W^{[L]}\_{ij}}
  $$
  
  Now, in component language, it is much simpler to identify where are all the matrix multiplications: The transpose of the gradient with respect to the cost function, multiplied by the jacobian matrix of the softmax activation (or whatever activation), and finally, the last derivative, which will evaluate to something depending on the activation of the previous layer.
  
  =].
  
  
  ---
  
  