I assume you’ve installed a local CUDA toolkit (with the nvcc CUDA compiler) and added cuDNN to it.
If so then virtual conda environments wouldn’t change anything as the CUDA toolkit it not a conda or pip package and should be visible in all envs (try to run which nvcc and it should point to e.g. /usr/local/cuda).
Note that a locally installed CUDA toolkit will only be used if you are building PyTorch from source or a custom CUDA extensions. The pip wheels and conda binaries ship with their own CUDA runtime and you would only need to install a sufficiently new NVIDIA driver to execute these binaries.