Here is how I understand this code…
h_grads = torch.autograd.grad(outputs=h_loss, inputs=cv_params, create_graph=True)

computes the gradients of h_loss w.r.t. cv_params and puts the gradients in h_grads. From my tests neither h_grad nor cv_param will have a grad_fn after this step, so I don’t see what graph gets created, nor where it is stored.
torch.autograd.backward(cv_params, h_grads)

Calculates the gradients of cv_params using h_grads as the initial gradients. I don’t see how this works. cv_params are not the results of computations, they are parameters of the model, so there is nothing to calculate. Oh I see, this simply adds h_grad to cv_param.grad.
Skipping a bit…
flat_params = []
for param in cv_params:
    flat_params.append(param.grad.view(-1))
flat_params = torch.cat(flat_params, 0)

This copies all cv_param.grad into a flat Variable. At this point, flat_params depends on param.grad for param in cv_params. So var_loss is simply the sum of the squares of param.grad, and these are copied into var_param.grad for var_param in var_params.
My understanding of all this is probably way off mark and I still can’t see where some bits of computation graph might be left hanging around.
Doing
for param in cv_params:
    param.detach()

at the end of each iteration might conceivably help, but I really have no idea.