Check Keep getting CUDA OOM error with Pytorch failing to allocate all free memory - PyTorch Forums 9 for the pytorch part of it.
I am seeing something similar for XLM, but in my case pytorch config override are not getting recognized. Need to check if hugging face is overriding it internally.