@albanD What’s the current recommended way to do this?
@zdevito Is there a way to preserve gradients through that? If so, what’s the right way to do it?
To illustrate my use case (meta-optimization):
import torch

def grad(f):
    def result(x):
        x_ = x.detach().requires_grad_(True) 
        f(x_).backward()
        return x_.grad
    return result

def f(x):
    return x**2

def grad_f(x):
    g1 = 2 * x
    g2 = grad(f)(x)
    assert (g1 == g2).all()
    print('g1: {!r}\ng2: {!r}'.format(g1, g2))
    return g1

def optimize(lr):
    x = torch.tensor(1.)
    for _ in range(10**2):
        x = x - lr * grad_f(x)
    return x

lr = torch.tensor(1e-3, requires_grad=True)
opt = torch.optim.SGD([lr], 1e-6, .9)
while True:
    loss = f(optimize(lr))
    print('{:6.4f} {:6.4f}'.format(loss.item(), lr.item()))
    opt.zero_grad()
    loss.backward()
    opt.step()

Note that, after the first step, g1 is connected to the computation graph defined by lr, but g2 is not:
g1: tensor(2.)
g2: tensor(2.)
g1: tensor(1.9960, grad_fn=<MulBackward0>)
g2: tensor(1.9960)
g1: tensor(1.9920, grad_fn=<MulBackward0>)
g2: tensor(1.9920)
g1: tensor(1.9880, grad_fn=<MulBackward0>)
g2: tensor(1.9880)
g1: tensor(1.9840, grad_fn=<MulBackward0>)
g2: tensor(1.9840)