The conda binaries and pip wheels ship with their own CUDA runtime and your local CUDA toolkit will not be used.
You would only need to provide a sufficiently new NVIDIA driver and it should work.
Your local CUDA toolkit would be used, if you are building PyTorch from source or are building custom CUDA extensions.