Deepspeed,
Deployment with multiple GPUs
To deploy this feature with multiple GPUs adjust the [Trainer] command line arguments as
following:
replace python -m torch.distributed.launch with deepspeed.
add a new argument --deepspeed ds_config.json, where ds_config.json is the DeepSpeed configuration file as
   documented here. The file naming is up to you.
Therefore, if your original command line looked as following:
bash
python -m torch.distributed.launch --nproc_per_node=2 your_program.py <normal cl args>
Now it should be:
bash
deepspeed --num_gpus=2 your_program.py <normal cl args> --deepspeed ds_config.json
Unlike, torch.distributed.launch where you have to specify how many GPUs to use with --nproc_per_node, with the
deepspeed launcher you don't have to use the corresponding --num_gpus if you want all of your GPUs used. The
