Normally when we’re doing backprop we would do the following:
loss.backward()  # This calculates the gradients
optimizer.step()   # This updates the net

However, what if I wish to accumulate the gradients? Meaning I want to run various loss.backward() multiple times first and accumulate the gradients before applying them in optimizer.step().

What do you recommend for storing the gradients? grads = loss.backward()?
How do I feed the accumulated gradients to optimizer.step?