Do you shuffle the data loader? In that case, it might be the same image that is too large that makes your training crash, but at different iterations since it’s not ordered…
The first intuition that comes to mind is that one (or multiple) image(s) is too large and computing gradients and stuff on GPU fills the memory, but the other images do not. If shuffling is enabled, I would try to disable it and see which image(s) make it crash, then resize those only and then check again (you will probably have to do it image by image).
If it’s not shuffled, I have absolutely no idea what could happen 
BTW, if not all your images are included in your small dataset, it’s totally possible that the small one doesn’t crash, if the too-large images are not present…