The cublas error might be raised, if you are running out of memory or if cublas runs indeed into an internal error.
If you made sure to have enough memory and reducing the batch size doesn’t solve it, you could create the cublas logs 1 and we can check them.



 Pedro_Domingues:

And also, since I want to get the most out of my 2 GPUs should I be using DistributedDataParallel instead? If so, do I need to produce major changes in my code?


Yes, you should use DDP for the best performance. This tutorial gives you an overview.