Could you plot the GPU memory usage along your script running time? There’s a torch.cuda function for that here 33. (In particular, it would be good to also reset the value at each iteration, as stated in the doc).
It’s weird though that it crashes at step 3-4 in your batch-size 50 run and not at all for your batch-size 40 run…