Answers.Body: <p>When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus.<br />
  If you have a function <span class="math-container">$$f:\mathbb{R}\rightarrow\mathbb{R}$$</span>
  the derivative is a function on its own and you have <span class="math-container">$$f':\mathbb{R}\rightarrow\mathbb{R}.$$</span>
  If you increase the dimension of the input space have
  <span class="math-container">$$f:\mathbb{R}^n\rightarrow\mathbb{R}.$$</span>
  The &quot;derivative&quot; in this case is called gradient and it is a vector collecting the <span class="math-container">$n$</span> partial derivatives of <span class="math-container">$f$</span>. The input space of the gradient function is <span class="math-container">$\mathbb{R}^n$</span> (the same as for <span class="math-container">$f$</span>), but the output is the collection of the <span class="math-container">$n$</span> derivatives, so the output space is also <span class="math-container">$\mathbb{R}^n$</span>. In other words
  <span class="math-container">$$\nabla f:\mathbb{R}^n\rightarrow\mathbb{R}^n,$$</span>
  which makes sense as for each point <span class="math-container">$x$</span> of the input space you get a vector (<span class="math-container">$\nabla f(x)$</span>) as output.<br />
  So far so good, but what happens if you consider a function that takes a vector as input and spits out a vector as output, i.e.
  <span class="math-container">$$f:\mathbb{R}^n\rightarrow\mathbb{R}^m?$$</span>
  How to compute the equivalent of the derivative? (This is the softmax case, where you have a vector as input and a vector as output.)<br />
  You can reduce this case to the previous case considering <span class="math-container">$f=(f_1, \dots, f_m)$</span>, where <span class="math-container">$f_i:\mathbb{R}^n\rightarrow\mathbb{R}.$</span> Now for each <span class="math-container">$f_i$</span> you can compute the gradient <span class="math-container">$\nabla f_i$</span> and end up with <span class="math-container">$m$</span> gradients. When you evaluate them at a point <span class="math-container">$x\in\mathbb{R}^n$</span> you get <span class="math-container">$m$</span> <span class="math-container">$n-$</span>dimensional vectors. These vector can be collected in a matrix, which is the Jacobian; formally
  <span class="math-container">$$Jf:\mathbb{R}^n\rightarrow\mathbb{R}^{m\times n}.$$</span>
  Finally, to answer your question, you get a Jacobian &quot;instead&quot;  of a gradient (they all represent the same concept) because the output of the softmax is not a single number but a vector.<br />
  By the way the sigmoid and relu are functions with one dimensional input and output, so they don't really have a gradient but a derivative. The trick is that people write <span class="math-container">$\sigma(W)$</span>, where <span class="math-container">$W$</span> is vector or a matrix, but they mean that <span class="math-container">$\sigma$</span> is applied component-wise, as a function from <span class="math-container">$\mathbb{R}$</span> to <span class="math-container">$\mathbb{R}$</span> (I know, it's confusing).<br />
  (I know, I kind of skipped your other question, but this answer was already long and I think you can convince yourself that the dimensions match correctly (with the Jacobian) for the update rule to work. If not I'll edit)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-05T12:00:32.387Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118054/discussion-on-answer-by-uskebasi-why-is-the-derivative-of-the-softmax-layer-shap).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-02T19:08:31.960Z
Answers.LastActivityDate: 2021-01-02T19:08:31.960Z
Answers.UserDisplayName: Uskebasi
Body: <p>If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than the output/softmax vector? Why is the shape of the softmax vector's derivative (the Jacobian) different than the shape of the derivative of the other activation functions, such as the ReLU and sigmoid?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-02T20:43:00.973Z
Comments.Text: [This blog post](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/) could also be useful.
Comments.UserDisplayName: nbro
Tags: <backpropagation><objective-functions><gradient-descent><linear-algebra><softmax>
Title: Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
UserDisplayName: Ilknur Mustafa
ThreadText: # Why is the derivative of the softmax layer shaped differently than the derivative of other neurons?
  
  (Asked by: username_0 on 2021-01-02)
  
  If the derivative is supposed to give the rate of change of a function at that point, then why is the derivative of the softmax layer (a vector) the Jacobian matrix, which has a different shape than the output/softmax vector? Why is the shape of the softmax vector's derivative (the Jacobian) different than the shape of the derivative of the other activation functions, such as the ReLU and sigmoid?
  
  **Tags:** backpropagation, objective-functions, gradient-descent, linear-algebra, softmax
  
  > **Comments:**
  > **username_1**: [This blog post](https://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/) could also be useful.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-01-02. Score: 3 | Accepted answer
  
  When you use the softmax activation function is usually as a last layer of your network and to get an output that is a vector. Now your confusion is about shapes, so let's review a bit of calculus.  
  If you have a function $$f:\mathbb{R}\rightarrow\mathbb{R}$$
  the derivative is a function on its own and you have $$f':\mathbb{R}\rightarrow\mathbb{R}.$$
  If you increase the dimension of the input space have
  $$f:\mathbb{R}^n\rightarrow\mathbb{R}.$$
  The "derivative" in this case is called gradient and it is a vector collecting the $n$ partial derivatives of $f$. The input space of the gradient function is $\mathbb{R}^n$ (the same as for $f$), but the output is the collection of the $n$ derivatives, so the output space is also $\mathbb{R}^n$. In other words
  $$\nabla f:\mathbb{R}^n\rightarrow\mathbb{R}^n,$$
  which makes sense as for each point $x$ of the input space you get a vector ($\nabla f(x)$) as output.  
  So far so good, but what happens if you consider a function that takes a vector as input and spits out a vector as output, i.e.
  $$f:\mathbb{R}^n\rightarrow\mathbb{R}^m?$$
  How to compute the equivalent of the derivative? (This is the softmax case, where you have a vector as input and a vector as output.)  
  You can reduce this case to the previous case considering $f=(f\_1, \dots, f\_m)$, where $f\_i:\mathbb{R}^n\rightarrow\mathbb{R}.$ Now for each $f\_i$ you can compute the gradient $\nabla f\_i$ and end up with $m$ gradients. When you evaluate them at a point $x\in\mathbb{R}^n$ you get $m$ $n-$dimensional vectors. These vector can be collected in a matrix, which is the Jacobian; formally
  $$Jf:\mathbb{R}^n\rightarrow\mathbb{R}^{m\times n}.$$
  Finally, to answer your question, you get a Jacobian "instead" of a gradient (they all represent the same concept) because the output of the softmax is not a single number but a vector.  
  By the way the sigmoid and relu are functions with one dimensional input and output, so they don't really have a gradient but a derivative. The trick is that people write $\sigma(W)$, where $W$ is vector or a matrix, but they mean that $\sigma$ is applied component-wise, as a function from $\mathbb{R}$ to $\mathbb{R}$ (I know, it's confusing).  
  (I know, I kind of skipped your other question, but this answer was already long and I think you can convince yourself that the dimensions match correctly (with the Jacobian) for the update rule to work. If not I'll edit)
  
  > **Comments:**
  > **username_1**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118054/discussion-on-answer-by-username_2-why-is-the-derivative-of-the-softmax-layer-shap).  
  
  ---
  
  