Answers.Body: <p>(ignoring the fact that is probably the other way around, meaning <span class="math-container">$z$</span> are the logits, <span class="math-container">$a = softmax(z)$</span>)</p>
  <p>Automatic differentiation expects you to take the derivative of a function <span class="math-container">$f: R^N \rightarrow R$</span>, where <span class="math-container">$f$</span> for example can be the loss function.</p>
  <p>If you have multiple outputs, you cannot apply GD, except if you &quot;collapse/aggregate&quot; these multiple outputs (for example, taking the mean)</p>
  <p>From an intuitive perspective, that matrix is telling you &quot;how to change <span class="math-container">$z$</span> so that <span class="math-container">$a$</span> increases&quot;... you have to decide which of these &quot;<span class="math-container">$a$</span>&quot; should increase, or, taking the mean of them, averaging over the gradients</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-10-23T14:47:51.827Z
Answers.LastActivityDate: 2024-10-23T14:47:51.827Z
Answers.UserDisplayName: Alberto
Body: <p>imaging input vector a = {a1,a2,a3}
  and z = softmax(a) = {z1,z2,z3}</p>
  <p>So, we expect than gradient of z with respect of a would be the same shape as vector a (so we can make gradient step: a = a - learning rate * dz/da). But the real shape is (3,3). So what should i do to make the gradient step? <a href="https://i.sstatic.net/g3uG12Iz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/g3uG12Iz.png" alt="enter image description here" /></a></p>
  
Tags: <gradient-descent>
Title: Softmax gradient for automatic differentiation
UserDisplayName: Тима 
ThreadText: # Softmax gradient for automatic differentiation
  
  (Asked by: username_0 on 2024-10-23)
  
  imaging input vector a = {a1,a2,a3}
  and z = softmax(a) = {z1,z2,z3}
  
  So, we expect than gradient of z with respect of a would be the same shape as vector a (so we can make gradient step: a = a - learning rate \* dz/da). But the real shape is (3,3). So what should i do to make the gradient step? [![enter image description here](https://i.sstatic.net/g3uG12Iz.png)](https://i.sstatic.net/g3uG12Iz.png)
  
  **Tags:** gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-10-23. Score: 1
  
  (ignoring the fact that is probably the other way around, meaning $z$ are the logits, $a = softmax(z)$)
  
  Automatic differentiation expects you to take the derivative of a function $f: R^N \rightarrow R$, where $f$ for example can be the loss function.
  
  If you have multiple outputs, you cannot apply GD, except if you "collapse/aggregate" these multiple outputs (for example, taking the mean)
  
  From an intuitive perspective, that matrix is telling you "how to change $z$ so that $a$ increases"... you have to decide which of these "$a$" should increase, or, taking the mean of them, averaging over the gradients
  
  
  ---
  
  