From PyTorch Cuda with anaconda not available 11 follows that you need to do a clean uninstall before you reinstall pytorch and attached packages. You might also check whether there are old files somewhere after the uninstall, see How can l uninstall PyTorch? 11.
If you want to install pytorch, you always need to install everything needed in one go, messing around with cuda as an additional package is error prone. It is recommended to install pytorch and additional packages in one go from conda:

Anaconda is our recommended package manager since it installs all dependencies.

And pip does not install all dependencies, when this post is right:

It uses preinstalled CUDA and doesn’t download own CUDA Toolkit.

See How to run pytorch with NVIDIA “cuda toolkit” version instead of the official conda “cudatoolkit” version? 31.
Therefore, using pip is your actual problem, because pip installed pytorch can be expected to need exactly the right version, else the dependencies can fail.
In short: do not use pip. Use conda with an independent “cudatoolkit”, do not use the system “cuda toolkit”, and that is also what pytorch recommends.