full details on how to configure various nodes and GPUs can be found here.
In fact, you can continue using -m torch.distributed.launch with DeepSpeed as long as you don't need to use
deepspeed launcher-specific arguments. Typically if you don't need a multi-node setup you're not required to use
the deepspeed launcher. But since in the DeepSpeed documentation it'll be used everywhere, for consistency we will
use it here as well.
Here is an example of running run_translation.py under DeepSpeed deploying all available GPUs:
bash
deepspeed examples/pytorch/translation/run_translation.py \
--deepspeed tests/deepspeed/ds_config_zero3.json \
--model_name_or_path t5-small --per_device_train_batch_size 1 \
--output_dir output_dir --overwrite_output_dir --fp16 \
--do_train --max_train_samples 500 --num_train_epochs 1 \
