Answers.Body: <p>I think your error is assuming that
  <span class="math-container">$$\frac{\partial y_1}{\partial x_1} \frac{\partial l}{\partial y_1} = \frac{\partial l}{\partial x_1}.$$</span>
  While 'cancelling' derivatives like this works sometimes, it is not true in general, and treating these quantities as fractions is not really correct in the usual formulation of calculus in terms of limits.</p>
  <p>If you run through the calculation using the <a href="https://en.wikipedia.org/wiki/Chain_rule" rel="nofollow noreferrer">chain rule</a> with Jacobian matrices, you should recover the correct product. If you're not clear on the compositions, write the functions <span class="math-container">$y_i(x_1, \dots, x_n)$</span> and <span class="math-container">$l(y_1, \dots, y_m)$</span>. If it helps define <span class="math-container">$y(x_1, \dots, x_n) = (y_1, \dots, y_m)$</span> then use the chain rule to compute <span class="math-container">$l(y(x_1, \dots, x_n))$</span>. If it would help to see the calculation in more detail, do let me know. Otherwise, it's a good exercise.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-03T21:40:29.237Z
Answers.LastActivityDate: 2021-08-03T21:40:29.237Z
Answers.UserDisplayName: htl
Body: <p>In the official PyTorch documentation there is the following calculation (<a href="https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html" rel="nofollow noreferrer">here</a>):</p>
  <p><span class="math-container">$$
  J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
  \frac{\partial y_{1}}{\partial x_{1}} &amp; \cdots &amp; \frac{\partial y_{m}}{\partial x_{1}} \\
  \vdots &amp; \ddots &amp; \vdots \\
  \frac{\partial y_{1}}{\partial x_{n}} &amp; \cdots &amp; \frac{\partial y_{m}}{\partial x_{n}}
  \end{array}\right)\left(\begin{array}{c}
  \frac{\partial l}{\partial y_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial y_{m}}
  \end{array}\right)=\left(\begin{array}{c}
  \frac{\partial l}{\partial x_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial x_{n}}
  \end{array}\right)
  $$</span></p>
  <p>However, I am wondering why the result isn't as follows:</p>
  <p><span class="math-container">$$
  J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
  \frac{\partial y_{1}}{\partial x_{1}} &amp; \cdots &amp; \frac{\partial y_{m}}{\partial x_{1}} \\
  \vdots &amp; \ddots &amp; \vdots \\
  \frac{\partial y_{1}}{\partial x_{n}} &amp; \cdots &amp; \frac{\partial y_{m}}{\partial x_{n}}
  \end{array}\right)\left(\begin{array}{c}
  \frac{\partial l}{\partial y_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial y_{m}}
  \end{array}\right)=\left(\begin{array}{c}
  m\frac{\partial l}{\partial x_{1}} \\
  \vdots \\
  m\frac{\partial l}{\partial x_{n}}
  \end{array}\right)
  $$</span></p>
  <p>As the matrix is multiplied by a vector and the <span class="math-container">$\partial y_{x}$</span> terms cancel out it is should be <span class="math-container">$m$</span> times <span class="math-container">$\frac{\partial l}{\partial x_{x}}$</span>.</p>
  <p>I know that the official PyTorch docs are probably right but I just can't get my head around why.</p>
  
Tags: <pytorch><vectors>
Title: Is this calculation of the vector-Jacobian product in the PyTorch documention wrong?
UserDisplayName: F.M.F.
ThreadText: # Is this calculation of the vector-Jacobian product in the PyTorch documention wrong?
  
  (Asked by: username_0 on 2021-08-03)
  
  In the official PyTorch documentation there is the following calculation ([here](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)):
  
  $$
  J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
  \frac{\partial y\_{1}}{\partial x\_{1}} & \cdots & \frac{\partial y\_{m}}{\partial x\_{1}} \\
  \vdots & \ddots & \vdots \\
  \frac{\partial y\_{1}}{\partial x\_{n}} & \cdots & \frac{\partial y\_{m}}{\partial x\_{n}}
  \end{array}\right)\left(\begin{array}{c}
  \frac{\partial l}{\partial y\_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial y\_{m}}
  \end{array}\right)=\left(\begin{array}{c}
  \frac{\partial l}{\partial x\_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial x\_{n}}
  \end{array}\right)
  $$
  
  However, I am wondering why the result isn't as follows:
  
  $$
  J^{T} \cdot \vec{v}=\left(\begin{array}{ccc}
  \frac{\partial y\_{1}}{\partial x\_{1}} & \cdots & \frac{\partial y\_{m}}{\partial x\_{1}} \\
  \vdots & \ddots & \vdots \\
  \frac{\partial y\_{1}}{\partial x\_{n}} & \cdots & \frac{\partial y\_{m}}{\partial x\_{n}}
  \end{array}\right)\left(\begin{array}{c}
  \frac{\partial l}{\partial y\_{1}} \\
  \vdots \\
  \frac{\partial l}{\partial y\_{m}}
  \end{array}\right)=\left(\begin{array}{c}
  m\frac{\partial l}{\partial x\_{1}} \\
  \vdots \\
  m\frac{\partial l}{\partial x\_{n}}
  \end{array}\right)
  $$
  
  As the matrix is multiplied by a vector and the $\partial y\_{x}$ terms cancel out it is should be $m$ times $\frac{\partial l}{\partial x\_{x}}$.
  
  I know that the official PyTorch docs are probably right but I just can't get my head around why.
  
  **Tags:** pytorch, vectors
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-03. Score: 1 | Accepted answer
  
  I think your error is assuming that
  $$\frac{\partial y\_1}{\partial x\_1} \frac{\partial l}{\partial y\_1} = \frac{\partial l}{\partial x\_1}.$$
  While 'cancelling' derivatives like this works sometimes, it is not true in general, and treating these quantities as fractions is not really correct in the usual formulation of calculus in terms of limits.
  
  If you run through the calculation using the [chain rule](https://en.wikipedia.org/wiki/Chain_rule) with Jacobian matrices, you should recover the correct product. If you're not clear on the compositions, write the functions $y\_i(x\_1, \dots, x\_n)$ and $l(y\_1, \dots, y\_m)$. If it helps define $y(x\_1, \dots, x\_n) = (y\_1, \dots, y\_m)$ then use the chain rule to compute $l(y(x\_1, \dots, x\_n))$. If it would help to see the calculation in more detail, do let me know. Otherwise, it's a good exercise.
  
  
  ---
  
  