Hi,
I would like to train a resnetxxx model on very large images (~3000x3000) in a domain where downscaling loses relevant information. My GPU only has memory enough for a single image at a time, so I look into gradient accumulation, discussed e.g.  here 1. This seems to work fine for the gradient itself, but is it possible to distribute the batch norm calculations over several (micro-)batches also?