Could you check if you might be running out of host memory and are seeing a massive slowdown due to the potential usage of swap space?
If so, limit the number of workers via MAX_JOBS=n python setup.py install.