```python
from transformers import T5ForConditionalGeneration, T5Config
import deepspeed
with deepspeed.zero.Init():
    config = T5Config.from_pretrained("t5-small")
    model = T5ForConditionalGeneration(config)
```
As you can see this gives you a randomly initialized model.
If you want to use a pretrained model, model_class.from_pretrained will activate this feature as long as
is_deepspeed_zero3_enabled() returns True, which currently is setup by the
[TrainingArguments] object if the passed DeepSpeed configuration file contains ZeRO-3 config
section. Thus you must create the [TrainingArguments] object before calling
from_pretrained. Here is an example of a possible sequence:
```python
from transformers import AutoModel, Trainer, TrainingArguments
training_args = TrainingArguments(..., deepspeed=ds_config)
