PyTorch conda binaries and pip wheels ship with their own CUDA (, cudnn, NCCL, etc.) runtime and your local CUDA toolkit will not be used unless you build PyTorch from source or any custom CUDA extension.
Yes, CUDA11 should work, if it supports your GPU architecture. If not, please create a topic here or an issue on GitHub and we’ll take a look.