I found /usr/lib/x86_64-linux-gnu/libcuda.so and created a symlink in /path_to_conda/miniconda3/envs/pytorch_build/lib64/stubs/ but as was to be expected this just led to another message
grep: /path_to_conda/miniconda3/envs/pytorch_build/version.txt: No such file or directory

And if I try to build pytorch I’m back at the original error, although which nvcc now yields the one within the conda environment.