Your local CUDA toolkit won’t be used, if you install the conda binaries or pip wheels, and you would only need to install the NVIDIA driver.
Since you are apparently running into env conflicts, you could try to create a new conda env and install the latest PyTorch version there.