Hi,
You will need to run as many backwards as gradients you want unfortunately.
The nice thing with sum/avg is that since you want a single gradient, you can compute them at once.
That being said, if your network has some special structure, you might be able to use tricks like this one to do it: https://github.com/cybertronai/autograd-hacks 4