What type of overhead does multiple calls to backward generate?

We need to go in cpp, discover the work that needs to be done, start GPU kernels for everything.
That takes some time.

omitted most of that (including zero_grad ). It’s related to differential privacy.

One thing I would say is that if the final transformation that you’re doing on the gradient is linear, then you can apply that same (similar) transformation to the loss before accumulation.
Otherwise, you can try to use something like this 2 to get a per-sample gradients in one backward pass (more or less) and then do all the post processing after that. That might be faster.