Could you lower the batch size to 1 and check, if your code works?
Also, could you check the output of nvidia-smi and see, how much memory your model uses and if other processes are using the GPU?
If your memory footprint is really large, you might try to use checkpointing 7 to trade compute for memory.