Thanks for the clarification. Usually you would get a CUDA error if you run OOM, so I was just wondering.
Do you see increased usage of your GPU memory in nvidia-smi?
I’m not sure if the usage of pin_memory might cause an OOM error, but just disable it and see if anything changes.