I would generally recommend to use a local CUDA, which matches the pre-built binaries to avoid CUDA version conflicts.
However, based on your last comment you are using the CUDA11.0 libtorch and a local CUDA11.1 to build your application?
Could you install CUDA11.0 locally or build libtorch with your local CUDA11.1?
Also, which GPU are you using?