Solved by alex.veuthey in post #2 
              
              
                Do you shuffle the data loader? In that case, it might be the same image that is too large that makes your training crash, but at different iterations since it’s not ordered… 
The first intuition that comes to mind is that one (or multiple) image(s) is too large and computing gradients and stuff on …