The conda binaries and pip wheels ship with their own CUDA (cudnn, NCCL, etc.) runtimes and your local installations won’t be used in built-in PyTorch methods.
If you need to build a custom CUDA extension, your local CUDA installation will be used and you should make sure the binaries use the same CUDA version as your local one.
Besides that your local libs will be used if you build PyTorch from source.