Answers.Body: <p>I didn't find the reference in the Goodfellow deep learning book. But here is how I derived it. Let all vectors be column vectors.</p>
  <p>The basic claim is that <span class="math-container">$\nabla_W F(u^\top W) = \nabla F(v^\top) u^\top$</span>, where <span class="math-container">$v^\top := u^\top W$</span>.</p>
  <p>Consider the simplest function <span class="math-container">$F(v^\top) := v^\top x$</span>. We can compute <span class="math-container">$\nabla_W (u^\top W x)$</span> either by looking up matrix differentiation rules, or by hand:</p>
  <p><span class="math-container">$$\partial_{W_{i,j}} u^\top W x = x_i u_j.$$</span>
  This shows that <span class="math-container">$\nabla_W (u^\top W x) = x u^\top$</span>.</p>
  <p>Finally convince yourself that general <span class="math-container">$F$</span> behaves the same way:
  <span class="math-container">$$\nabla_W F(u^\top W) = \nabla_W (u^\top W \nabla F(v^\top)) = \nabla F(v^\top) u^\top,$$</span>
  since the post-multiplication by <span class="math-container">$\nabla F(v^\top)$</span> spans all linear maps taking <span class="math-container">$u^\top W$</span> to <span class="math-container">$\mathbb{R}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-01T07:13:43.777Z
Answers.LastActivityDate: 2022-05-01T07:13:43.777Z
Answers.UserDisplayName: John Jiang
Body: <p>The paper: <a href="https://arxiv.org/abs/2110.11309" rel="nofollow noreferrer">https://arxiv.org/abs/2110.11309</a>, makes the following claim at the end of page 3:</p>
  <blockquote>
  <p>The gradient of loss <span class="math-container">$L$</span> with respect to weights <span class="math-container">$W_l$</span> of an MLP is a rank-1 matrix for each of B batch elements <span class="math-container">$\nabla_{w_l}L = \sum_{i=1}^B \delta_{l+1}^i {u_l^i}^T$</span>, where <span class="math-container">$\delta_{l+1}^i$</span> is the gradient of the loss for batch element <span class="math-container">$i$</span> with respect to the preactivations at layer <span class="math-container">$l + 1$</span>, and <span class="math-container">${u_l^i}^T$</span> are the inputs to layer <span class="math-container">$l$</span> for batch element i.</p>
  </blockquote>
  <p>Suppose that we have an MLP with <span class="math-container">$k$</span> hidden layers (every hidden layer is followed by an activation function). Then the weight matrices will be <span class="math-container">$W_1, W_2, \dots, W_k$</span> (plus the biases, but they are irrelevant for now), and their sizes will be <span class="math-container">$(D_1, D), (D_2, D_1), \dots (D_k, D_{k-1})$</span> correspondingly, where <span class="math-container">$D$</span> is the number of input features.</p>
  <p>Therefore, hidden layer <span class="math-container">$l$</span> has a weight matrix <span class="math-container">$W_l$</span> of size <span class="math-container">$(D_l, D_{l-1})$</span>. Its gradient wrt the loss (for 1 batch element), <span class="math-container">$\frac{\partial L}{\partial W_l}$</span>, will also be a matrix of size <span class="math-container">$(D_l, D_{l-1})$</span>.</p>
  <p>So if I understand correctly, the authors of the paper are claiming that <span class="math-container">$\frac{\partial L}{\partial W_l}$</span> is a rank-1 matrix? That is, every row (or column) can be expressed as a linear combination of 1 only row (or column)? If yes, why? How?</p>
  
Tags: <weights><multilayer-perceptrons><gradient>
Title: Rank of gradient-of-loss with respect to layer weights in an MLP
UserDisplayName: Andrew
ThreadText: # Rank of gradient-of-loss with respect to layer weights in an MLP
  
  (Asked by: username_0 on 2021-11-25)
  
  The paper: <https://arxiv.org/abs/2110.11309>, makes the following claim at the end of page 3:
  
  > The gradient of loss $L$ with respect to weights $W\_l$ of an MLP is a rank-1 matrix for each of B batch elements $\nabla\_{w\_l}L = \sum\_{i=1}^B \delta\_{l+1}^i {u\_l^i}^T$, where $\delta\_{l+1}^i$ is the gradient of the loss for batch element $i$ with respect to the preactivations at layer $l + 1$, and ${u\_l^i}^T$ are the inputs to layer $l$ for batch element i.
  
  Suppose that we have an MLP with $k$ hidden layers (every hidden layer is followed by an activation function). Then the weight matrices will be $W\_1, W\_2, \dots, W\_k$ (plus the biases, but they are irrelevant for now), and their sizes will be $(D\_1, D), (D\_2, D\_1), \dots (D\_k, D\_{k-1})$ correspondingly, where $D$ is the number of input features.
  
  Therefore, hidden layer $l$ has a weight matrix $W\_l$ of size $(D\_l, D\_{l-1})$. Its gradient wrt the loss (for 1 batch element), $\frac{\partial L}{\partial W\_l}$, will also be a matrix of size $(D\_l, D\_{l-1})$.
  
  So if I understand correctly, the authors of the paper are claiming that $\frac{\partial L}{\partial W\_l}$ is a rank-1 matrix? That is, every row (or column) can be expressed as a linear combination of 1 only row (or column)? If yes, why? How?
  
  **Tags:** weights, multilayer-perceptrons, gradient
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-05-01. Score: 1
  
  I didn't find the reference in the Goodfellow deep learning book. But here is how I derived it. Let all vectors be column vectors.
  
  The basic claim is that $\nabla\_W F(u^\top W) = \nabla F(v^\top) u^\top$, where $v^\top := u^\top W$.
  
  Consider the simplest function $F(v^\top) := v^\top x$. We can compute $\nabla\_W (u^\top W x)$ either by looking up matrix differentiation rules, or by hand:
  
  $$\partial\_{W\_{i,j}} u^\top W x = x\_i u\_j.$$
  This shows that $\nabla\_W (u^\top W x) = x u^\top$.
  
  Finally convince yourself that general $F$ behaves the same way:
  $$\nabla\_W F(u^\top W) = \nabla\_W (u^\top W \nabla F(v^\top)) = \nabla F(v^\top) u^\top,$$
  since the post-multiplication by $\nabla F(v^\top)$ spans all linear maps taking $u^\top W$ to $\mathbb{R}$.
  
  
  ---
  
  