So you’re trying to make them work separately as well as “in combination”?
One thing you could try is to make the find_package conditional on if (NOT TARGET torch_library) or somesuch to have it executed exactly once.
If that doesn’t solve it, maybe it would help if you showed a bit about your CMake setup.
Best regards
Thomas