Yes, your explanation is right. The gradients from each backward pass would be summed to the previous gradients.
You could use it to artificially increase your batch size.
If you don’t have enough GPU memory, but need a larger batch size, you could sum the gradients for several smaller batches, average them, and finally perform the optimization step.
Ah ok, then you don’t need the Variable wrapper anymore, since they were merged with tensors.