So I narrowed down the problem. It seems like there’s a problem when I do the last grad update with inputs  where baseline_params is part of the network and etas are defined as:
for _ in cv_params:
    eta = 2 * F.sigmoid(Variable(torch.zeros(1), requires_grad=True))
    if opts['cuda']:
        eta = 2 * F.sigmoid(Variable(torch.zeros(1), requires_grad=True).cuda())
    eta_params.append(nn.Parameter(eta.data))
    etas.append(eta)

I actually found a bug where I was doing this:
var_params = baseline_params + etas
instead of
var_params = baseline_params + eta_params
But changing to this gives an error:
RuntimeError: One of the differentiated Variables appears to not have been used in the graph
If I change the last 2 lines to:
var_grads = torch.autograd.grad(outputs=var_loss, inputs=baseline_params, create_graph=True)
torch.autograd.backward(baseline_params, var_grads)

It works fine.