If after trying everything suggested you still encounter build issues, please, proceed with the GitHub Issue of
FairScale.
Usage:
To use the first version of Sharded data-parallelism, add --sharded_ddp simple to the command line arguments, and
make sure you have added the distributed launcher -m torch.distributed.launch --nproc_per_node=NUMBER_OF_GPUS_YOU_HAVE if you haven't been using it already.
For example here is how you could use it for run_translation.py with 2 GPUs:
bash
python -m torch.distributed.launch --nproc_per_node=2 examples/pytorch/translation/run_translation.py \
--model_name_or_path t5-small --per_device_train_batch_size 1   \
--output_dir output_dir --overwrite_output_dir \
--do_train --max_train_samples 500 --num_train_epochs 1 \
--dataset_name wmt16 --dataset_config "ro-en" \
