Does nvidia-smi reports a cuda version? It only reports the driver version from what I remember. Cuda is independent of that.
You can set CUDA_HOME before running the pytorch install to point to the conda install. But you should not need to do that…