Hello @aitutakiv, I am a bit confused. How should I compute the gradients wrt to the original weights? If it’s possible, can you review my code. It seems I am approaching the problem quite naively, and missing out on something important! It would be great if you are able to give me a few pointers 
My inner loop code:
def inner_loop(learner, batch, loss_fn, steps, device):
    X, y = batch # Batch Size = 2*shots*ways
    X, y = X.to(device), y.to(device)

    # Separate data into adaptation/evalutation sets
    # Dtrain and Dtest both of size ways*shots
    adapt_idx = np.zeros(X.size(0), dtype=bool)
    adapt_idx[np.arange(1*5) * 2] = True # shots*ways
    eval_idx = torch.from_numpy(~adapt_idx)
    apt_idx = torch.from_numpy(adapt_idx)
    adapt_data, adapt_labels = X[adapt_idx], y[adapt_idx]
    eval_data, eval_labels = X[eval_idx], y[eval_idx]

    # Adaptation Steps (Steps: 0 to K-1)
    for steps in range(0, steps): # steps : no of adaption steps
        pred = learner(adapt_data)
        loss = loss_fn(pred, adapt_labels)
        grad = torch.autograd.grad(loss, learner.parameters())
        new_weight = list(map(lambda p: p[1] - 0.4*p[0], zip(grad, learner.parameters()))) # theta' = theta - alpha*grads
        with torch.no_grad(): # Updating the new_weights to learner
            for i, params in enumerate(learner.parameters()):
                params.copy_(new_weight[i])
        
    # Evaluation loss, accuracy
    pred_val = learner(eval_data)
    loss_val = loss_fn(pred_val, eval_labels)
    pred_val = pred_val.argmax(dim=1).view(eval_labels.shape)
    acc_val = (pred_val == eval_labels).sum().float()/eval_labels.size(0)
    
    return loss_val, acc_val

And the outer loop:
iterations = 5
meta_batch_size = 32
for iteration in range(iterations):
    train_loss, train_acc = 0, 0
    for task in range(meta_batch_size):
        # Copy model parameters. 
        learner = deepcopy(new_model)
        batch = tasksets.train.sample()  # Generates 2*shots*ways samples.
        # The batch will be split into Dtrain and Dtest in the inner loop (both of size shots*ways)
        loss_val, acc_val = inner_loop(learner, batch, loss_fn, 1, device) # Gradient steps: 1
        train_loss += loss_val/meta_batch_size
        train_acc += acc_val/meta_batch_size
        del learner
    meta_opt.zero_grad()
    train_loss.backward()
    meta_opt.step()
    if iteration % 1 == 0:
        print('Training Accuracy:', train_acc.item())
        print('Training Loss:', train_loss.item())

As you might have guessed, the computation graph is breaking in the outer loop, thus train_loss is not updating any weights. It would be awesome if you are able to point out a few things! Thank you!
Just for the sake of additional information, I will add the Train Loss, Acc here:
Training Accuracy: 0.3437499403953552
Training Loss: 1.4747799634933472
Training Accuracy: 0.3062499761581421
Training Loss: 1.510756254196167
Training Accuracy: 0.38749998807907104
Training Loss: 1.5028401613235474
Training Accuracy: 0.3312499225139618
Training Loss: 1.4998550415039062
Training Accuracy: 0.35625001788139343
Training Loss: 1.477844476699829