I have a similar question. I have two sets of parameters, and I would like to do something like co-ordinate descent.
x = Variable(torch.randn(10, 20), requires_grad=False)  # data
u = Variable(torch.randn(10, 5), requires_grad=True)  # param set 1
v = Variable(torch.randn(5, 20), requires_grad=True)  # param set 2

opt_u = optim.Adagrad([u])
opt_v = optim.Adagrad([v])

for i in range(10):

    opt_u.zero_grad()
    loss = -torch.sum(x * (u @ v))
    loss.backward()  # gives me grads w.r.t  u and v. I need only grads w.r.t  'u'
    opt_u.step()  # update only param set 1 (u), keeping set 2 fixed.

    opt_v.zero_grad()
    loss = -torch.sum(x * (u @ v))
    loss.backward()  # gives me grads w.r.t  u and v. I need only grads w.r.t 'v'
    opt_v.step()  # update only param set 2 (v), keeping set 1 fixed.

I would like pytorch to avoid computing gradients w.r.t some leafs.
Is it possible? If yes, will it provide any speed improvements?