Did you build PyTorch/Caffe2 before from source and was it working correctly?
If so, did you use the same CUDA version?
Your setup looks correct, so could you maybe clean the build and rerun it?
If that doesn’t help, could you check, if you would be able to build libtorch in a docker container?