I am also facing this issue. I want to use libtorch on a cluster, where CUDA is not installed in /usr/.
@Miles_Cranmer , I followed your workaround, and I got to compile my small example. It also works with CUDA.
There is a problem with your last post though, you say you edit the same file twice. I think the second file you are referring to is link.txt. I found it using the command:
grep -nr /usr/local/cuda/

In link.txt, I removed the arguments (or parts of arguments) that referred to this path, and it worked.