I work with @Deepali and have been looking at this as well. I think I see what’s happening now…
The libcuda.so that’s being loaded at run time is the stub version that comes with CUDA Toolkit. That’s suitable for dynamic linking at build time, but isn’t actually functional.
$ LD_DEBUG=all ./libtest 2>&1 | grep 'object=.*libcuda[.]so'
     95995:     object=/opt/anaconda3/envs/test_env/lib64/stubs/libcuda.so.1 [0]
     95995:     object=/opt/anaconda3/envs/test_env/lib64/stubs/libcuda.so.1 [0] 

And that library is being found because of the RPATH in the built binary:
$ objdump -p libtest | grep RPATH
  RPATH                [...]:/opt/anaconda3/envs/test_env/lib64/stubs:[...]

Unfortunately RPATH takes precedence over LD_LIBRARY_PATH :
$ LD_LIBARY_PATH=/usr/lib64 ./libtest
device_count: 0
Is available? 0

But LD_PRELOAD would get around the problem:
$ LD_PRELOAD=/usr/lib64/libcuda.so.440.33.01 ./libtest
device_count: 4
Is available? 1

So I think the trick will be to find a way to prevent the CUDA Toolkit stubs/ library directory from being added to RPATH.
I’m not sure whether it’s something specific to our environment (we’re building in a conda setup) that causes that, or if it would be something in the general libtorch stuff.
I tried just not installing the CUDA Toolkit development stuff (leaving only the runtime installed), but then the Torch cmake find_package stuff fails:
CUDA_TOOLKIT_ROOT_DIR not found or specified                                                                                                                                       
-- Could NOT find CUDA (missing: CUDA_TOOLKIT_ROOT_DIR CUDA_NVCC_EXECUTABLE CUDA_INCLUDE_DIRS)

I didn’t look into that yet, but assume it’s probably looking for nvcc to figure out where CUDA’s installed.