Training with wrong GPU count
Hi,
Thank you for your repo :)

I'm fine-tuning with 4 GPU (run_squad, bert model)
And I found that gpu count is wrong when to do distributed training.
I've got 1 GPU count and that's caused by source code below
Is there any reason to set n_gpu = 1 when to do distributed training?

 if args.local_rank == -1 or args.no_cuda:
        device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
        n_gpu = torch.cuda.device_count()
    else:
        torch.cuda.set_device(args.local_rank)
        device = torch.device("cuda", args.local_rank)
        n_gpu = 1 <= this!!!
        # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
        torch.distributed.init_process_group(backend='nccl')

Yes, this is expected behavior. Each script in distributed training has ownership over one GPU only.

You can read this blog post for details on parallel and distributed training: https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255