I ran
grep -nr /usr/local/cuda

inside the libtorch source directory, and I got these results:
share/cmake/Caffe2/Caffe2Targets.cmake:82:  INTERFACE_LINK_LIBRARIES "caffe2::cudart;c10_cuda;caffe2;caffe2::cufft;caffe2::curand;caffe2::cudnn;/usr/local/cuda/lib64/libculibos.a;dl;/usr/local/cuda/lib64/libculibos.a;caffe2::cublas"
share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake:34:# ``CUDA_BIN_PATH=/usr/local/cuda1.0`` instead of the default
share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake:35:# ``/usr/local/cuda``) or set ``CUDA_TOOLKIT_ROOT_DIR`` after configuring.  If
share/cmake/Caffe2/Modules_CUDA_fix/upstream/FindCUDA.cmake:939:    list(APPEND CUDA_LIBRARIES -Wl,-rpath,/usr/local/cuda/lib)
share/cmake/Gloo/GlooTargets.cmake:58:  INTERFACE_LINK_LIBRARIES "/usr/local/cuda/lib64/libcudart.so;\$<LINK_ONLY:pthread>"
share/cmake/Gloo/GlooTargets.cmake:74:  INTERFACE_LINK_LIBRARIES "/usr/local/cuda/lib64/libcudart.so;gloo;/pytorch/build/nccl/lib/libnccl_static.a;dl;rt"
Binary file lib/libcaffe2_gpu.so matches

I think this might be a problem, since not everybody has CUDA installed under /usr/.