to no avail, the next thing to try is to pre-build the modules before installing them.
To make a local build for DeepSpeed:
bash
git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="8.6" DS_BUILD_CPU_ADAM=1 DS_BUILD_UTILS=1 pip install . \
--global-option="build_ext" --global-option="-j8" --no-cache -v \
--disable-pip-version-check 2>&1 | tee build.log
If you intend to use NVMe offload you will need to also include DS_BUILD_AIO=1 in the instructions above (and also
install libaio-dev system-wide).
Edit TORCH_CUDA_ARCH_LIST to insert the code for the architectures of the GPU cards you intend to use. Assuming all
your cards are the same you can get the arch via:
bash
CUDA_VISIBLE_DEVICES=0 python -c "import torch; print(torch.cuda.get_device_capability())"
