@albanD, I  have some doubt about what the computation graph looks like in the case that we accumulate the loss manually.
Assume there are total 256 batches for the dataset.  For the normal case (case1), for each batch in an epoch, a new computation graph is created and after the backward pass, the graph is freed. So 256 computation graphs are created and freed during one epoch.
In this case(case2), since we only do backward on every 64 batches. Does that mean only 4 graphs are created?  The graph are created by composing 64 smaller graphs in case1 and the root node in the bigger graph is total_loss. The 64 smaller graph all have the same set of learnable parameters. If that is the case, the bigger graph will consume a lot of memory since it have 64 copies of the small graph.
Is that right? Do you have any ideas?