I’m really not sure about how the internals are handled, but it would make sense if it now crashes on the same image at the same location every time, if the memory is allocated in the order of execution.
For example, your network could have three layers, PyTorch tries to allocate GPU memory for the first layer, given some input size X, then the second layer, but then the third layer cannot allocate memory because X is too large and the first two layers have almost filled the memory…
You could try resizing the image that crashes your program to smaller and smaller sizes and see if the error comes up at the same location, and this may also help you determine what is the maximum size your GPU can handle with that network architecture!