🤗 Transformers integrates DeepSpeed via 2 options:
Integration of the core DeepSpeed features via [Trainer]. This is everything done for you type
   of integration - just supply your custom config file or use our template and you have nothing else to do. Most of
   this document is focused on this feature.
If you don't use [Trainer] and want to use your own Trainer where you integrated DeepSpeed
   yourself, core functionality functions like from_pretrained and from_config include integration of essential
   parts of DeepSpeed like zero.Init for ZeRO stage 3 and higher. To tap into this feature read the docs on
   deepspeed-non-trainer-integration.
What is integrated:
Training:
DeepSpeed ZeRO training supports the full ZeRO stages 1, 2 and 3 with ZeRO-Infinity (CPU and NVME offload).
Inference:
