Btorch:

and it’s weird because the difference from the normal run is saving a small loss tensor, which is negligable.


That’s not the case, since each tensor chunk would also store the entire computation graph, so you would use the same memory in the end.
You could simulate the larger batch size using one of the approaches described in this post 2.