Okay, I think I should have provided more details in the sense that you think I didn’t look elsewhere, so here it is.
The issue comes when the Trainer tries to use cpu_adam extension. Now there are two ways, JIT way of building extension on fly, and prebuilding the extensions. JIT way and prebuilding fails because of the error trace related to pybind11. So I search for cpu_adam, I land upon this issue 2. Now this issue is open, you provided some details which I did follow. Trainer docs 1, local rank doesn’t need to configured as you said. I did set PATH and LD_LIBRARY_PATH as well.
I did see this 1 also, where in I tried pre-building:


git clone https://github.com/microsoft/DeepSpeed/
cd DeepSpeed
rm -rf build
TORCH_CUDA_ARCH_LIST="6.1;8.6" DS_BUILD_OPS=1 python setup.py build_ext -j8 bdist_wheel

This doesn’t seem to work for the same reason. Looked into 1, 2 as well.