both configured to offload to cpu. If you have NVMe, experiment with offloading to NVMe if you're running under
ZeRO-3.
Work is being done to enable estimating how much memory is needed for a specific model: PR.
Notes
DeepSpeed works with the PyTorch [Trainer] but not TF [TFTrainer].
While DeepSpeed has a pip installable PyPI package, it is highly recommended that it gets installed from source to best match your hardware and also if you need to enable
  certain features, like 1-bit Adam, which aren't available in the pypi distribution.
You don't have to use the [Trainer] to use DeepSpeed with 🤗 Transformers - you can use any model
  with your own trainer, and you will have to adapt the latter according to the DeepSpeed integration instructions.
Non-Trainer Deepspeed Integration
The [~deepspeed.HfDeepSpeedConfig] is used to integrate Deepspeed into the 🤗 Transformers core
