Your 3080 needs CUDA>=11.0, so you have to install the PyTorch binaries with 11.3 or 11.5.
And yes, you are correct that the PyTorch binaries ship with their own CUDA runtime and do not use the locally installed CUDA toolkit unless you build PyTorch from source or a custom CUDA extension.