The conda binary will only install the CUDA runtime in the current conda environment, not a full CUDA toolkit in /usr/local/cuda.
After installing it, you can check the CUDA version used in the PyTorch binaries via:
print(torch.version.cuda)