If I hardcode the

set(CMAKE_CUDA_COMPILER_LIBRARY_ROOT "/opt/soft/cuda/11.4.2-1")


in CMakeDetermineCUDACompiler.cmake to force it past that point just to see what breaks, it fails the nvcc “simple test program” precheck. Manually repeating that precheck succeeds.  I don’t get this unless the cuda install is relocated so… bad cuda relocation?  Both /usr/local and /opt/soft/cuda/{version}/ cuda installs successfully build and run their samples. Same thing if both installs coexist or are on different login node images, so it’s not an intersect. The 11.4.2-1 was prepackage, bet there’s something wrong with it’s install despite the successful samples. Either that or something with the combo of 11.4.2-1 and python 3.10.0 for the pytorch build?
What install method for your cuda, the one in /workspace/src? Tarball, “raw” installer, package, module, other?
What python?