Toward Training Trillion Parameter Models, by Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He.
This provided support is new and experimental as of this writing.
CUDA Extension Installation Notes
As of this writing, both FairScale and Deepspeed require compilation of CUDA C++ code, before they can be used.
While all installation issues should be dealt with through the corresponding GitHub Issues of FairScale and Deepspeed, there are a few common issues that one may encounter while building
any PyTorch extension that needs to build CUDA extensions.
Therefore, if you encounter a CUDA-related build issue while doing one of the following or both:
bash
pip install fairscale
pip install deepspeed
please, read the following notes first.
In these notes we give examples for what to do when pytorch has been built with CUDA 10.2. If your situation is
