Answers.Body: <p><span class="math-container">$\nabla_{\theta_{i-1}} \theta_{i-1} = \mathbf{I}$</span> in a similar way that <span class="math-container">$\frac{d f}{dx} = 1$</span> for <span class="math-container">$f(x) = x$</span>. Strictly speaking, <span class="math-container">$\mathbf{I}$</span> should be a vector of <span class="math-container">$1s$</span> with the same dimensionality as <span class="math-container">$\theta_{i-1}$</span>, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of <span class="math-container">$\theta_{i-1}^j$</span> with respect to <span class="math-container">$\theta_{i-1}^k$</span>, for all <span class="math-container">$k$</span>, for all <span class="math-container">$j$</span>, which will make up an identity matrix.</p>
  
  <p>Regarding your first question, <span class="math-container">$\nabla_{\theta} \theta_{0}$</span> probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because <span class="math-container">$\nabla_{\theta} \theta_{0}$</span> actually means <span class="math-container">$\nabla_{\theta_0} \theta_{0}$</span>. I would need to dive into it.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T16:00:50.617Z
Answers.Text: @DuttaA Well, I guess that what you say is consistent with my explanation. If you look at that part of the formula, you see terms where the gradient with respect to the previous parameters is taken of the current parameters. However, initially, we only have $\theta_0$ (i.e. no previous parameters), so the derivative of $\theta_0$ with respect to itself is $1$.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-10T09:42:35.883Z
Answers.Text: I updated my question with a second solution. Please have a look and give some comment about it. It seems like in this case we don't need to worry about the $\theta_{0}$  anymore.
Answers.UserDisplayName: Long
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-03-09T15:19:07.230Z
Answers.LastActivityDate: 2020-03-09T15:28:17.760Z
Answers.LastEditDate: 2020-03-09T15:28:17.760Z
Answers.UserDisplayName: nbro
Body: <p>According to the authors of <a href="https://arxiv.org/pdf/1703.03400.pdf" rel="nofollow noreferrer">this paper</a>, to improve the performance, they decided to </p>
  
  <blockquote>
    <p>drop backward pass and using a first-order approximation</p>
  </blockquote>
  
  <p>I found <a href="https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html" rel="nofollow noreferrer">a blog</a> which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):</p>
  
  <ol>
  <li>Why 
  <img src="https://latex.codecogs.com/gif.latex?%5Cnabla_%7B%5Ctheta%7D%5Ctheta_%7B0%7D" alt=""> disappeared in the next line.</li>
  <li>How come <img src="https://latex.codecogs.com/gif.latex?%5Cnabla_%7B%5Ctheta_%7Bi-1%7D%7D%5Ctheta_%7Bi-1%7D%3DI" alt=""> (which is an Identity matrix)</li>
  </ol>
  
  <p><a href="https://i.sstatic.net/7aXRX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/7aXRX.png" alt="FOMAML"></a></p>
  
  <p><strong>Update:</strong> I also found <a href="https://www.slideshare.net/ssuser62b35f/introduction-to-maml-model-agnostic-meta-learning-with-discussions-124492943" rel="nofollow noreferrer">another math solution</a> for this. To me it looks less intuitive but there's no confusion with the disappearance of  as in the first solution.
  <a href="https://i.sstatic.net/kEFn7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kEFn7.png" alt="first order MAML"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-09T15:22:11.733Z
Comments.Text: Regarding your 1st question, what is $\theta$? In all other cases, $\theta$ has a subscript.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-10T09:40:43.410Z
Comments.Text: θ was in the Algorithm 1, but then in the blog it uses both θ and $\theta_{0}$. Maybe they're both "initial model parameter" and the notation was not consistent. I'm working on understanding it too ..
Comments.UserDisplayName: Long
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-02T03:50:54.470Z
Comments.Text: @Long what is the rationality behind ignoring the 2nd-order derivative in FOMAML and regarding it as an identity matrix?
Comments.UserDisplayName: S.EB
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-07T04:11:22.627Z
Comments.Text: @S.EB it is common in many research that people ignore high-order derivatives to avoid computation overhead and simpler implementation. See [this](https://lilianweng.github.io/posts/2018-11-30-meta-learning/)
  
  `The meta-optimization step above relies on second derivatives. To make the computation less expensive, a modified version of MAML omits second derivatives, resulting in a simplified and cheaper implementation, known as First-Order MAML (FOMAML)`
Comments.UserDisplayName: Long
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-12T04:32:11.830Z
Comments.Text: @Long Thank you very much for your explanation. I have two questions: 1) why meta-learning is using two steps of gradients, also called bi-level optimization? 2) why it is said gradients of gradients? what is the exact meaning of it?
Comments.UserDisplayName: S.EB
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-19T17:10:18.380Z
Comments.Text: @S.EB I think these questions should be in a [new post](https://ai.stackexchange.com/questions/ask), the comment section will not be beneficial to those who have similar questions as you do. I will try to answer them and I think there are many people with better knowledge (than I have) would do so too.
Comments.UserDisplayName: Long
Tags: <deep-learning><math><gradient-descent><meta-learning><model-agnostic-meta-learning>
Title: Understanding the derivation of the first-order model-agnostic meta-learning
UserDisplayName: Long
ThreadText: # Understanding the derivation of the first-order model-agnostic meta-learning
  
  (Asked by: username_0 on 2020-03-09)
  
  According to the authors of [this paper](https://arxiv.org/pdf/1703.03400.pdf), to improve the performance, they decided to
  
  > drop backward pass and using a first-order approximation
  
  I found [a blog](https://lilianweng.github.io/lil-log/2018/11/30/meta-learning.html) which discussed how to derive the math but got stuck along the way (please refer to the embedded image below):
  
  1. Why
     ![](https://latex.codecogs.com/gif.latex?%5Cnabla_%7B%5Ctheta%7D%5Ctheta_%7B0%7D) disappeared in the next line.
  2. How come ![](https://latex.codecogs.com/gif.latex?%5Cnabla_%7B%5Ctheta_%7Bi-1%7D%7D%5Ctheta_%7Bi-1%7D%3DI) (which is an Identity matrix)
  
  [![FOMAML](https://i.sstatic.net/7aXRX.png)](https://i.sstatic.net/7aXRX.png)
  
  **Update:** I also found [another math solution](https://www.slideshare.net/ssuser62b35f/introduction-to-maml-model-agnostic-meta-learning-with-discussions-124492943) for this. To me it looks less intuitive but there's no confusion with the disappearance of as in the first solution.
  [![first order MAML](https://i.sstatic.net/kEFn7.png)](https://i.sstatic.net/kEFn7.png)
  
  **Tags:** deep-learning, math, gradient-descent, meta-learning, model-agnostic-meta-learning
  > **Comments:**
  > **username_1**: Regarding your 1st question, what is $\theta$? In all other cases, $\theta$ has a subscript.  
  > **username_0**: θ was in the Algorithm 1, but then in the blog it uses both θ and $\theta_{0}$. Maybe they're both "initial model parameter" and the notation was not consistent. I'm working on understanding it too ..  
  > **username_2**: @Long what is the rationality behind ignoring the 2nd-order derivative in FOMAML and regarding it as an identity matrix?  
  > **username_0**: @S.EB it is common in many research that people ignore high-order derivatives to avoid computation overhead and simpler implementation. See [this](https://lilianweng.github.io/posts/2018-11-30-meta-learning/)  `The meta-optimization step above relies on second derivatives. To make the computation less expensive, a modified version of MAML omits second derivatives, resulting in a simplified and cheaper implementation, known as First-Order MAML (FOMAML)`  
  > **username_2**: @Long Thank you very much for your explanation. I have two questions: 1) why meta-learning is using two steps of gradients, also called bi-level optimization? 2) why it is said gradients of gradients? what is the exact meaning of it?  
  > **username_0**: @S.EB I think these questions should be in a [new post](https://ai.stackexchange.com/questions/ask), the comment section will not be beneficial to those who have similar questions as you do. I will try to answer them and I think there are many people with better knowledge (than I have) would do so too.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-03-09. Score: 2
  
  $\nabla\_{\theta\_{i-1}} \theta\_{i-1} = \mathbf{I}$ in a similar way that $\frac{d f}{dx} = 1$ for $f(x) = x$. Strictly speaking, $\mathbf{I}$ should be a vector of $1s$ with the same dimensionality as $\theta\_{i-1}$, but they are probably abusing notation here and putting such a vector as the diagonal elements of a matrix. Alternatively (actually, the most likely reason!), they are computing the partial derivative of $\theta\_{i-1}^j$ with respect to $\theta\_{i-1}^k$, for all $k$, for all $j$, which will make up an identity matrix.
  
  Regarding your first question, $\nabla\_{\theta} \theta\_{0}$ probably becomes 1, but I am not familiar enough with the math of this paper to tell you why. Maybe it's because $\nabla\_{\theta} \theta\_{0}$ actually means $\nabla\_{\theta\_0} \theta\_{0}$. I would need to dive into it.
  
  > **Comments:**
  > **username_1**: @DuttaA Well, I guess that what you say is consistent with my explanation. If you look at that part of the formula, you see terms where the gradient with respect to the previous parameters is taken of the current parameters. However, initially, we only have $\theta_0$ (i.e. no previous parameters), so the derivative of $\theta_0$ with respect to itself is $1$.  
  > **username_0**: I updated my question with a second solution. Please have a look and give some comment about it. It seems like in this case we don't need to worry about the $\theta_{0}$  anymore.  
  
  ---
  
  