Scratch that, I found the problem.  I had my Header Search Paths set to recursive and that caused my issues.
For anyone that finds this… here’s what I did to get this installed on WatchOS.

cloned the pytorch repo
cd pytorch
git checkout v1.4.0
git submodule sync
git submodule update --init --recursive
BUILD_PYTORCH_MOBILE=1 IOS_PLATFORM=WATCHOS scripts/build_ios.sh
add files from build_ios/install/include/ and build_ios/install/lib/ to my XCode project
in XCode target build settings, add the include directory to Header Search Paths, make sure it’s non-recursive.
in XCode target build settings, Other Linker Flags,  -force_load “path to lib directory”

That should do it!