sajastu:

It appears that the trainer class of transformers automatically handles the multi gpu training when the GPU devices are known (i.e., CUDA_VISIBLE_DEVICES flag).


Could you link to the trainer code, which would implement this logic?
Based on the output of nvidia-smi it doesn’t seem to be the case or work as intended.