@ptrblck any idea? the tl;dr is that I can survive OOM events in a none distributed setting (just discarding this training minibatch, cleaning up memory and continuing), but using distributed setting I can’t.
This is important for me as I’m in medical imaging setting in which resolution is important, and cropping is too destructive.
Any suggestions and/or pointing me to relevant people if necessary is very welcome