Note we can’t really restore from accumulated gradients with batch>1 because we lose all the batch-related info.
so the question should be for batch=1.
Currently I did not find any way to do this (automatically) with Pytorch.