Hi, those equations are an example of how you do MAML with two steps of gradient descent in one task. f is the loss function. The first two lines are the two steps of gradient descent for the task objective. The third line is the gradient descent for the meta objective.
To compare it with MAML in Algorithm 1 on its paper,

The first two lines are the Line 6 of the Algorithm 1 (because we do two steps of gradient descent just so you can imagine if you want to implement general “n-steps of inner gradient descent” version of MAML).
The third line is the Line 8 of the Algorithm 1. However, in the code, I stop at only computing the df(theta_2)/dtheta and store that value in a list. This list will contain the df(theta_2)/dtheta of all tasks at the end. So, after exiting the for data in dataset loop, I can average that list and fo the meta-update as in the Line 8. The list of df(theta_2)/dtheta is like the copy of the model you are asking for.


For example is it possible/easy to store a copy of the model where you zero all parameters then apply the weight updates from the real model to that model one task at a time.

Feel free to ask/discuss further if you still have doubts as my answer might not fully resolve your question.
Regarding parallelization across multiple GPUs, I am not sure on how to implement it. I think it can be done efficiently by computing the meta-gradient of task on separate GPUs. Then each gradients computed on separate GPUs are sent to the main-GPU to be averaged and then used to update the model. Then, the model on each separate GPUs is updated by the main-GPU. Basically it’s like a synchronous update. (Sorry for the bad explanation, I have little experiences on this so I don’t know much about the terminology)
Ha, I found this article 1! We can do data parallelism + centarlized synchronous update.