hey @valgi0 my suggestion would be to try out the new accelerate library: GitHub - huggingface/accelerate: 🚀 A simple way to train and use PyTorch models with multi-GPU, TPU, mixed-precision 153
in particular, there is an nlp example that shows you how to configure accelerate for the multi-GPU case here: accelerate/examples at main · huggingface/accelerate · GitHub 75