Some deeper digging seems to suggest that this might be related to this issue: Libtorch: Segmentation fault when running torch::jit::load · Issue #49460 · pytorch/pytorch · GitHub
If we don’t want to assume that’s the source of the problem, the following are guesses:
CMake and find_package will only find the shared libraries during the build.  PyTorch + CUDA also needs to be able to find the DLL files.  I can’t think any simple ways I would recommend.  You could copy the DLLs into the directory with the executable, but they’re huge.
Another thing to check would be if you’re building in DEBUG or RELEASE mode.  If you’re building in release mode but have debug DLLs from PyTorch installed, that can cause issues.  I ran into that once.