Hi @HyeyeonKoo
for multi-GPU training you need to launch the script using torch.distributed.launch, have a look at this section of example docs
  
      
      github.com
  
  
    

huggingface/transformers - Distributed training and mixed precision 103

master/examples

  🤗Transformers: State-of-the-art Natural Language Processing for Pytorch and TensorFlow 2.0.