you should open your terminal and execute nvidia-smi to see the actual use of your GPU. If you still have free memory include torch.cuda.empty_cache() in your code before loading the model on GPU. Finally reducing batch_size could be a way to solve it.