Or maybe should this be a feature request for torch.autograd.backward?

What would be the feature?
Note that you don’t have to do retain_grad, you can pass inputs= to backward to get the .grad field only for these Tensors.
Also I’m curious what is the benefit of doing this compared to fw/bw or the smaller batches and accumulating the gradients?