Your local CUDA toolkit won’t be used, if you install the conda binaries or pip wheels, as they ship with their own CUDA runtime. You would only need to install the appropriate NVIDIA driver.
However, your local CUDA installation will be used if you build PyTorch from source or build custom CUDA extensions.
I would not recommend to try to isolate the seg fault, as it might have been a known issue which was already fixed.
For the GLIBC issue: could you create a new virtual environment and try to install the binaries again?