Answers.Body: <p>In short, <strong>the Jacobian matrix is a generalization of the gradient for vector-valued functions</strong>. </p>
  
  <p>Recall that the gradient is a vector of partial derivatives of a multi-variable function. So, consider a multi-variable function of the form <span class="math-container">$f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}$</span>. The output of this function is <span class="math-container">$f(x_1, x_2, \dots, x_N) = y$</span>, where <span class="math-container">$x_i \in \mathcal{X}_i$</span>, for <span class="math-container">$i=1, \dots, N$</span> and <span class="math-container">$y \in \mathcal{Y}$</span>. And the gradient is <span class="math-container">$\nabla f = \left[ \frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_N} \right] \in \mathbb{R}$</span>. </p>
  
  <p>A <a href="https://en.wikipedia.org/wiki/Vector-valued_function" rel="nofollow noreferrer">vector-valued function</a> is a function whose output is a vector, i.e. a function of the form <span class="math-container">$f: \mathcal{X} \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M$</span> (I am not sure if this notation is rigorous enough!), so the output of this function is a vector <span class="math-container">$f(x) = [y_1, y_2, \dots, y_M]$</span>, where <span class="math-container">$x \in \mathcal{X}$</span> and <span class="math-container">$y_i \in \mathcal{Y}_i$</span>, for <span class="math-container">$i = 1, \dots, M$</span>. You can also view a vector-valued function <span class="math-container">$f$</span> as a vector of scalar-valued functions <span class="math-container">$[f_1, f_2, \dots, f_M]$</span>, where <span class="math-container">$f_i: \mathcal{X} \rightarrow \mathcal{Y}_i$</span>, for all <span class="math-container">$i$</span>.</p>
  
  <p>You can also have multi-variable vector-valued functions, i.e. functions of the form </p>
  
  <p><span class="math-container">$$f: \mathcal{X}_1 \times \mathcal{X}_2 \times \dots \times \mathcal{X}_N \rightarrow \mathcal{Y}_1 \times \mathcal{Y}_2 \times \dots \times \mathcal{Y}_M.$$</span></p>
  
  <p>The Jacobian matrix is an <span class="math-container">$N \times M$</span> matrix with one partial derivative for each combination of inputs and outputs (i.e. <span class="math-container">$f_i$</span>). </p>
  
  <p>If you want to optimize a multi-variable vector-valued function, you can make use of the Jacobian, in a similar way that you make use of the gradient in the case of multi-variable functions, but, although I've seen it in the past, I can't provide now a concrete example of an application of the Jacobian (but <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2020-lecture04-neuralnets.pdf#page=9" rel="nofollow noreferrer">the linked slides</a> probably do that).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-13T15:38:49.637Z
Answers.LastActivityDate: 2020-05-13T18:08:03.590Z
Answers.LastEditDate: 2020-05-13T18:08:03.590Z
Answers.UserDisplayName: nbro
Body: <p>I came across these slides <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2020-lecture04-neuralnets.pdf" rel="nofollow noreferrer">Natural Language Processing with Deep Learning CS224N/Ling284</a>, in the context of natural language processing, which talk about the Jacobian as a generalization of the gradient.</p>
  
  <p>I know there is a lot of topic regarding this on the internet, and trust me, I've googled it. But things are getting more and more confused for me.</p>
  
  <p>In simple words, how is the Jacobian a generalization of the gradient? How can it be used in gradient descent?</p>
  
Tags: <deep-learning><natural-language-processing><math><gradient-descent>
Title: How is the Jacobian a generalisation of the gradient?
UserDisplayName: Pluviophile
ThreadText: # How is the Jacobian a generalisation of the gradient?
  
  (Asked by: username_0 on 2020-05-13)
  
  I came across these slides [Natural Language Processing with Deep Learning CS224N/Ling284](http://web.stanford.edu/class/cs224n/slides/cs224n-2020-lecture04-neuralnets.pdf), in the context of natural language processing, which talk about the Jacobian as a generalization of the gradient.
  
  I know there is a lot of topic regarding this on the internet, and trust me, I've googled it. But things are getting more and more confused for me.
  
  In simple words, how is the Jacobian a generalization of the gradient? How can it be used in gradient descent?
  
  **Tags:** deep-learning, natural-language-processing, math, gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-13. Score: 1 | Accepted answer
  
  In short, **the Jacobian matrix is a generalization of the gradient for vector-valued functions**.
  
  Recall that the gradient is a vector of partial derivatives of a multi-variable function. So, consider a multi-variable function of the form $f: \mathcal{X}\_1 \times \mathcal{X}\_2 \times \dots \times \mathcal{X}\_N \rightarrow \mathcal{Y}$. The output of this function is $f(x\_1, x\_2, \dots, x\_N) = y$, where $x\_i \in \mathcal{X}\_i$, for $i=1, \dots, N$ and $y \in \mathcal{Y}$. And the gradient is $\nabla f = \left[ \frac{\partial f}{\partial x\_1}, \dots, \frac{\partial f}{\partial x\_N} \right] \in \mathbb{R}$.
  
  A [vector-valued function](https://en.wikipedia.org/wiki/Vector-valued_function) is a function whose output is a vector, i.e. a function of the form $f: \mathcal{X} \rightarrow \mathcal{Y}\_1 \times \mathcal{Y}\_2 \times \dots \times \mathcal{Y}\_M$ (I am not sure if this notation is rigorous enough!), so the output of this function is a vector $f(x) = [y\_1, y\_2, \dots, y\_M]$, where $x \in \mathcal{X}$ and $y\_i \in \mathcal{Y}\_i$, for $i = 1, \dots, M$. You can also view a vector-valued function $f$ as a vector of scalar-valued functions $[f\_1, f\_2, \dots, f\_M]$, where $f\_i: \mathcal{X} \rightarrow \mathcal{Y}\_i$, for all $i$.
  
  You can also have multi-variable vector-valued functions, i.e. functions of the form
  
  $$f: \mathcal{X}\_1 \times \mathcal{X}\_2 \times \dots \times \mathcal{X}\_N \rightarrow \mathcal{Y}\_1 \times \mathcal{Y}\_2 \times \dots \times \mathcal{Y}\_M.$$
  
  The Jacobian matrix is an $N \times M$ matrix with one partial derivative for each combination of inputs and outputs (i.e. $f\_i$).
  
  If you want to optimize a multi-variable vector-valued function, you can make use of the Jacobian, in a similar way that you make use of the gradient in the case of multi-variable functions, but, although I've seen it in the past, I can't provide now a concrete example of an application of the Jacobian (but [the linked slides](http://web.stanford.edu/class/cs224n/slides/cs224n-2020-lecture04-neuralnets.pdf#page=9) probably do that).
  
  
  ---
  
  