I have a similar doubt. What I have found so far is that nn.Parameters are LEAF nodes with no history. So, the Gradient Descent (GD) operations that you are trying to perform for the inner loop (adapted_params[key] = val - meta_step_size * grad) won’t be recorded in the computation graph.
The following may help you understand why updating model parameters like the way you have won’t work in PyTorch in more detail:

  
    
    
    Cannot calculate second order gradients even though `create_graph=True` 
  
  
    I have a training loop iteration that looks like this… 
mu, logvar = m(x)

alpha = torch.zeros(size, requires_grad=True)
loss = alpha * criterion(mu, logvar)

loss.backward(retain_graph=True, create_graph=True)
for p in m.parameters():
  p = p - LR * p.grad

x_next, y_next = data[j + 1]
mu_next, logvar_next = m(x_next)

meta_loss = criterion(mu_next, logvar_next)
grad_alpha = torch.autograd.grad(meta_loss, alpha)[0]

and pytorch is complaining that RuntimeError: One of the differentiated Tensors…
  


In fact, you may want to use the package “HIGHER” or refer to the following codes for a workaround (look for MetaModule in the repositories):

https://github.com/AdrienLE 10
https://github.com/xjtushujun/meta-weight-net/blob/bd1fd3e297c59df7c5264fb636c67d8ee03bcf0d/resnet.py#L240 12

I hope that helps you understand the problem with your method.