Could you support other distributed training backends from command line, I'm using GLOO.
https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/src/transformers/training_args.py#L858

Seems like it's hard-coded to NCCL. Could you instead make this an argument? 

@sgugger 
You can initialize the backend you want in your script. Accelerate won't re-initialize it in this case.<|||||>@sgugger Thanks, I was following the examples provided in examples/pytorch/question-answering/run_qa.py. 
As in 
https://github.com/huggingface/transformers/blob/76c4d8bf26de3e4ab23b8afeed68479c2bbd9cbd/examples/pytorch/question-answering/run_qa.py#L212, it will parse from all the listed allowed arguments including `TrainingArguments`
I think there will be no arguments provided to choose my backend here. And I think it won't use Accelerate anyway. So, in this case, it seems the only way to change it to 'gloo' backend is modifying the source code in training_args.py?<|||||>Oh I was confused on where you opened the issue. There is no way to change the backend in the `Trainer` code, as it's not just one argument that could be setup differently. You should use Accelerate for a more customized training loop.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>OK, I just made a PR here to further justify what I meant: https://github.com/huggingface/transformers/pull/14012, please take a look. Make any suggestions or corrections as you need. Thanks.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.<|||||>AFAIK, this is handled by the xpu_backend `training_args` parameter passed into the `Trainer` class now. Making a note here for future reference.
See https://github.com/huggingface/transformers/blob/ee8e80a060d65ab349743ffcb5842365eb0e5606/src/transformers/training_args.py#L315-L316 for more details