DeepSpeed ZeRO Inference supports ZeRO stage 3 with ZeRO-Infinity. It uses the same ZeRO protocol as training, but
   it doesn't use an optimizer and a lr scheduler and only stage 3 is relevant. For more details see:
   deepspeed-zero-inference.
There is also DeepSpeed Inference - this is a totally different technology which uses Tensor Parallelism instead of
ZeRO (coming soon).
Trainer Deepspeed Integration
Installation
Install the library via pypi:
bash
pip install deepspeed
or via transformers' extras:
bash
pip install transformers[deepspeed]
or find more details on the DeepSpeed's GitHub page and
advanced install.
If you're still struggling with the build, first make sure to read zero-install-notes.
If you don't prebuild the extensions and rely on them to be built at run time and you tried all of the above solutions
