It shouldn’t impact any system-wide CUDA installs.
In fact I’m using a lot of different conda environments with different CUDA (and PyTorch) versions besides a system CUDA install for building PyTorch from source. So far I didn’t get into trouble.