Deepspeed ZeRO-Infinity (deepspeed==0.3.15) has been just integrated. You need to use the transformers master branch to use it.
There are 2 important changes that you need to be aware of if you’re already using DeepSpeed integration in transformers:


After this release only config params that are set to auto will get automatically overriden/set to the correct/recommended values, everything else is left as is. This is to avoid the previously confusing behavior of never being quite sure what gets overridden and what not despite the logger telling what it did override. The new behavior is completely unambiguous.
See examples

zero2 6
zero3 8

Full doc: Trainer — transformers 4.5.0.dev0 documentation 8


If you are using massive models and aren’t using example scripts, make sure to read:
Full doc: Trainer — transformers 4.5.0.dev0 documentation 6


Everything else should work as before or better.
The docs were revamped a lot too - if you find anything unclear or lacking please let me know.
You probably want to install deepspeed master though, since 0.3.15 left some debug prints in-place, which creates a lot of noise, which has been fixed in master. So:
pip install git+https://github.com/microsoft/DeepSpeed

If you encounter any problems please post an Issue and tag @stas00 to it. Thank you!