Thanks for the code!
Could you lower the batch sizes of both DataLoaders and check the memory usage on your GPU?
It would be interesting to see the peak usage, and where it occurs.
Usually it’s not a bad idea to keep some “spare” memory on your GPU so that small fluctuations won’t cause an OOM.