Answers.Body: <p>According to: &quot;<a href="https://arxiv.org/abs/1703.03400" rel="nofollow noreferrer">Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks</a>&quot; it is initialized in <a href="https://github.com/cbfinn/maml/blob/a7f45f1bcd7457fe97b227a21e89b8a82cc5fa49/maml.py#L17" rel="nofollow noreferrer">Class MAML</a>:</p>
  <pre><code>class MAML:
      def __init__(self, dim_input=1, dim_output=1, test_num_updates=5):
          &quot;&quot;&quot; must call construct_model() after initializing MAML! &quot;&quot;&quot;
          self.dim_input = dim_input
          self.dim_output = dim_output
          self.update_lr = FLAGS.update_lr
          self.meta_lr = tf.placeholder_with_default(FLAGS.meta_lr, ())
          self.classification = False
          self.test_num_updates = test_num_updates
          if FLAGS.datasource == 'sinusoid':
              self.dim_hidden = [40, 40]
              self.loss_func = mse
              self.forward = self.forward_fc
              self.construct_weights = self.construct_fc_weights
          elif FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
              self.loss_func = xent
              self.classification = True
              if FLAGS.conv:
                  self.dim_hidden = FLAGS.num_filters
                  self.forward = self.forward_conv
                  self.construct_weights = self.construct_conv_weights
              else:
                  self.dim_hidden = [256, 128, 64, 64]
                  self.forward=self.forward_fc
                  self.construct_weights = self.construct_fc_weights
              if FLAGS.datasource == 'miniimagenet':
                  self.channels = 3
              else:
                  self.channels = 1
              self.img_size = int(np.sqrt(self.dim_input/self.channels))
          else:
              raise ValueError('Unrecognized data source.')
  </code></pre>
  <blockquote>
  <p>&quot;How does MAML inner loop optimization work?&quot;</p>
  </blockquote>
  <p>Where Rectified Linear Unit (ReLU) neural networks are locally almost linear (<a href="https://arxiv.org/abs/1412.6572" rel="nofollow noreferrer">Goodfellow et al.</a>,
  2015), and second derivatives close
  to zero, using a first-order approximation removes the need for computing Hessian-vector products in an additional backward pass.</p>
  <p>It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in &quot;<a href="https://www.semanticscholar.org/paper/Understanding-the-difficulty-of-training-deep-Glorot-Bengio/b71ac1e9fb49420d13e084ac67254a0bbd40f83f" rel="nofollow noreferrer">Understanding the difficulty of training deep feedforward neural networks</a>&quot;, Xavier Glorot and Yoshua Bengio, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.</p>
  <p>A method to alleviate the risk of overfitting for gradient-based meta-learning is explained in: &quot;<a href="https://arxiv.org/abs/2004.05859" rel="nofollow noreferrer">Regularizing Meta-Learning via Gradient Dropout</a>&quot; - April 13 2020, by Tseng, Chen, Tsai, Liu, Lin, and Yang:</p>
  <blockquote>
  <p>&quot;[they] aim to find model parameters that
  are sensitive to changes in the task, such that small changes
  in the parameters will produce large improvements on the
  loss function of any task drawn from <span class="math-container">$p(T)$</span>, when altered in
  the direction of the gradient of that loss.&quot;.</p>
  </blockquote>
  <p>Combined, those methods shorten the time taken to complete each pass.</p>
  <p>In the line:</p>
  <pre><code>FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
              self.loss_func = xent
  </code></pre>
  <p>See pages 6 and 7:</p>
  <blockquote>
  <p>&quot;A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient
  at the post-update parameter values <span class="math-container">$θ^
  0_i$</span>, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the
  improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.&quot;.<br />
  <span class="math-container">$$\theta \leftarrow \theta \; - \beta \nabla_\theta \sum_{\mathcal{T}_i\thicksim p (\mathcal{T})} \mathcal{L_{T_i}}(\mathcal{f}_{\theta^{{'}}_i}) \tag{1}
  $$</span></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-29T13:03:20.477Z
Answers.Text: Hi, thank you for the answer, unfortunately, I am not sure that I understand. In the code, you have sent me I can see that loss for each set of parameters in the inner gradient is passed to the outer loop however I still don't understand how it is backpropagated to the original $\theta$ . Derivations are handled by TensorFlow (I guess that it uses some clever tricks to avoid the Hessian) but I don't understand the principle...
Answers.UserDisplayName: Grumpy C
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-29T14:11:33.923Z
Answers.Text: @GrumpyC, that is described separately for omniglot and miniimagenet on pages 6&7 of the paper you cited; using a first-order approximation.
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-29T14:21:39.860Z
Answers.Text: A different paper than the one you mentioned: "[ES-MAML: Simple Hessian-Free Meta Learning](https://arxiv.org/abs/1910.01215)" is clearer, backpropagation isn't used.
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-29T20:30:33.130Z
Answers.Text: The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi-layer neural networks. Actually, back-propagation refers only to **one** method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient.
  
  — Page 204, Deep Learning, 2016.
  
  It would be fair to say that a neural network is trained or learns using Stochastic Gradient Descent as a shorthand, as it is assumed that the back-propagation algorithm is used to calculate gradients as part of the optimization procedure.
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-31T04:52:50.343Z
Answers.Text: For a simplified explanation of the activation function see here: https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/ and https://en.wikipedia.org/wiki/Activation_function
Answers.UserDisplayName: Rob
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-05-28T21:48:04.940Z
Answers.LastActivityDate: 2022-05-29T10:58:49.987Z
Answers.LastEditDate: 2022-05-29T10:58:49.987Z
Answers.UserDisplayName: Rob
Body: <p>I started to learn meta-learning, reading the MAML paper <a href="https://arxiv.org/pdf/1703.03400.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/1703.03400.pdf</a></p>
  <p><a href="https://i.sstatic.net/kXMXz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/kXMXz.png" alt="enter image description here" /></a></p>
  <p>In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of inner SGD.
  I will calculate adapted parameters after two or more gradient steps (<span class="math-container">$\theta{'}$</span>, <span class="math-container">$\theta^{''}$</span> , <span class="math-container">$\theta^{n}$</span>), then using testing parameters, I will have a loss in respect to the original <span class="math-container">$\theta$</span> (If I understand the derivation correctly). Now I am supposed to backpropagate through the gradient. Unfortunately, I am not sure how it is done...</p>
  <blockquote>
  <p><em>Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
  Page 3.</em>
  <br> &quot;The MAML meta-gradient update involves a gradient
  through a gradient&quot;</p>
  </blockquote>
  <p>To do this, I understand that I have to store each <span class="math-container">$\theta^{n}$</span>, but don't know how the loss from <span class="math-container">$\theta^{n}$</span> to <span class="math-container">$\theta^{n-1}$</span> is transferred up to the original  <span class="math-container">$\theta$</span>.<br>
  I guess that the for the last series of parameters (<span class="math-container">$\theta^{n}$</span>) loss is calculated in the standard way with the testing set, but then I somehow need to pass information saying how much the previous set of parameters was wrong... <em>(Gradient of the gradient?)</em></p>
  <p>I see Hessian and vector products popping up everywhere on the internet, but I cannot imagine how that works, and have no idea how it is calculated and passed/implemented...</p>
  <p>Can someone explain to me how the inner loop [back-propagation trough meta-gradient] is working - how the derivations go and how loss is transferred/updated?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-29T11:20:36.413Z
Comments.Text: I see too many questions in this post. Edit your post to leave only one specific question.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-05-29T12:56:32.473Z
Comments.Text: Hi, sorry for the confusion, I have clarified the question, and I hope that it is now more readable.
Comments.UserDisplayName: Grumpy C
Tags: <optimization><gradient-descent><gradient><meta-learning><model-agnostic-meta-learning>
Title: How does MAML inner loop optimization works?
UserDisplayName: Grumpy C
ThreadText: # How does MAML inner loop optimization works?
  
  (Asked by: username_0 on 2022-05-28)
  
  I started to learn meta-learning, reading the MAML paper <https://arxiv.org/pdf/1703.03400.pdf>
  
  [![enter image description here](https://i.sstatic.net/kXMXz.png)](https://i.sstatic.net/kXMXz.png)
  
  In the inner loop, I am calculating adapted parameters for each task, I will be doing multiple steps of inner SGD.
  I will calculate adapted parameters after two or more gradient steps ($\theta{'}$, $\theta^{''}$ , $\theta^{n}$), then using testing parameters, I will have a loss in respect to the original $\theta$ (If I understand the derivation correctly). Now I am supposed to backpropagate through the gradient. Unfortunately, I am not sure how it is done...
  
  > *Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
  > Page 3.*
  >   
  >  "The MAML meta-gradient update involves a gradient
  > through a gradient"
  
  To do this, I understand that I have to store each $\theta^{n}$, but don't know how the loss from $\theta^{n}$ to $\theta^{n-1}$ is transferred up to the original $\theta$.  
  I guess that the for the last series of parameters ($\theta^{n}$) loss is calculated in the standard way with the testing set, but then I somehow need to pass information saying how much the previous set of parameters was wrong... *(Gradient of the gradient?)*
  
  I see Hessian and vector products popping up everywhere on the internet, but I cannot imagine how that works, and have no idea how it is calculated and passed/implemented...
  
  Can someone explain to me how the inner loop [back-propagation trough meta-gradient] is working - how the derivations go and how loss is transferred/updated?
  
  **Tags:** optimization, gradient-descent, gradient, meta-learning, model-agnostic-meta-learning
  > **Comments:**
  > **username_1**: I see too many questions in this post. Edit your post to leave only one specific question.  
  > **username_0**: Hi, sorry for the confusion, I have clarified the question, and I hope that it is now more readable.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-05-28. Score: 0
  
  According to: "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks" it is initialized in Class MAML:
  class MAML:
      def __init__(self, dim_input=1, dim_output=1, test_num_updates=5):
          """ must call construct_model() after initializing MAML! """
          self.dim_input = dim_input
          self.dim_output = dim_output
          self.update_lr = FLAGS.update_lr
          self.meta_lr = tf.placeholder_with_default(FLAGS.meta_lr, ())
          self.classification = False
          self.test_num_updates = test_num_updates
          if FLAGS.datasource == 'sinusoid':
              self.dim_hidden = [40, 40]
              self.loss_func = mse
              self.forward = self.forward_fc
              self.construct_weights = self.construct_fc_weights
          elif FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
              self.loss_func = xent
              self.classification = True
              if FLAGS.conv:
                  self.dim_hidden = FLAGS.num_filters
                  self.forward = self.forward_conv
                  self.construct_weights = self.construct_conv_weights
              else:
                  self.dim_hidden = [256, 128, 64, 64]
                  self.forward=self.forward_fc
                  self.construct_weights = self.construct_fc_weights
              if FLAGS.datasource == 'miniimagenet':
                  self.channels = 3
              else:
                  self.channels = 1
              self.img_size = int(np.sqrt(self.dim_input/self.channels))
          else:
              raise ValueError('Unrecognized data source.')
  
  
  "How does MAML inner loop optimization work?"
  
  Where Rectified Linear Unit (ReLU) neural networks are locally almost linear (Goodfellow et al.,
  2015), and second derivatives close
  to zero, using a first-order approximation removes the need for computing Hessian-vector products in an additional backward pass.
  It is initialized by xavier_initializer_conv2d from TensorFlow, as explained in "Understanding the difficulty of training deep feedforward neural networks", Xavier Glorot and Yoshua Bengio, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256, 2010.
  A method to alleviate the risk of overfitting for gradient-based meta-learning is explained in: "Regularizing Meta-Learning via Gradient Dropout" - April 13 2020, by Tseng, Chen, Tsai, Liu, Lin, and Yang:
  
  "[they] aim to find model parameters that
  are sensitive to changes in the task, such that small changes
  in the parameters will produce large improvements on the
  loss function of any task drawn from $p(T)$, when altered in
  the direction of the gradient of that loss.".
  
  Combined, those methods shorten the time taken to complete each pass.
  In the line:
  FLAGS.datasource == 'omniglot' or FLAGS.datasource == 'miniimagenet':
              self.loss_func = xent
  
  See pages 6 and 7:
  
  "A significant computational expense in MAML comes.from the use of second derivatives when backpropagating the meta-gradient through the gradient operator in the meta-objective (see Equation (1)). On MiniImagenet, we show a comparison to a first-order approximation of MAML, where these second derivatives are omitted. Note that the resulting method still computes the meta-gradient
  at the post-update parameter values $θ^
  0_i$, which provides for effective meta-learning. Surprisingly however, the performance of this method is nearly the same as that obtained with full second derivatives, suggesting that most of the
  improvement in MAML comes from the gradients of the objective at the post-update parameter values, rather than the second order updates from differentiating through the gradient update.".
  $$\theta \leftarrow \theta \; - \beta \nabla_\theta \sum_{\mathcal{T}_i\thicksim p (\mathcal{T})} \mathcal{L_{T_i}}(\mathcal{f}_{\theta^{{'}}_i}) \tag{1}
  $$
  
  > **Comments:**
  > **username_0**: Hi, thank you for the answer, unfortunately, I am not sure that I understand. In the code, you have sent me I can see that loss for each set of parameters in the inner gradient is passed to the outer loop however I still don't understand how it is backpropagated to the original $\theta$ . Derivations are handled by TensorFlow (I guess that it uses some clever tricks to avoid the Hessian) but I don't understand the principle...  
  > **username_2**: @GrumpyC, that is described separately for omniglot and miniimagenet on pages 6&7 of the paper you cited; using a first-order approximation.  
  > **username_2**: A different paper than the one you mentioned: "[ES-MAML: Simple Hessian-Free Meta Learning](https://arxiv.org/abs/1910.01215)" is clearer, backpropagation isn't used.  
  > **username_2**: The term back-propagation is often misunderstood as meaning the whole learning algorithm for multi-layer neural networks. Actually, back-propagation refers only to **one** method for computing the gradient, while another algorithm, such as stochastic gradient descent, is used to perform learning using this gradient.  — Page 204, Deep Learning, 2016.  It would be fair to say that a neural network is trained or learns using Stochastic Gradient Descent as a shorthand, as it is assumed that the back-propagation algorithm is used to calculate gradients as part of the optimization procedure.  
  > **username_2**: For a simplified explanation of the activation function see here: https://machinelearningmastery.com/choose-an-activation-function-for-deep-learning/ and https://en.wikipedia.org/wiki/Activation_function  
  
  ---
  
  