model = AutoModel.from_pretrained("t5-small")
trainer = Trainer(model=model, args=training_args, ...)
```
If you're using the official example scripts and your command line arguments include --deepspeed ds_config.json
with ZeRO-3 config enabled, then everything is already done for you, since this is how example scripts are written.
Note: If the fp16 weights of the model can't fit onto the memory of a single GPU this feature must be used.
For full details on this method and other related features please refer to Constructing Massive Models.
Also when loading fp16-pretrained models, you will want to tell from_pretrained to use
torch_dtype=torch.float16. For details, please, see from_pretrained-torch-dtype.
Gathering Parameters
Under ZeRO-3 on multiple GPUs no single GPU has all the parameters unless it's the parameters for the currently
