The conda binaries and pip wheels ship with their CUDA (cudnn, NCCL, etc.) runtime, so you don’t need a local CUDA installation to use native PyTorch operations.
However, you would have to install a matching CUDA version, if you want to build PyTorch from source or build custom CUDA extensions.