For some reason, my xcconfig for test didn’t have the following line:
LIBRARY_SEARCH_PATHS = $(inherited) "${PODS_ROOT}/LibTorch/install/lib"
It was also missing some OTHER_LDFLAGS like pytorch_qnnpack.
Not really sure why or how this fixed it, but I pasted those lines in. There is some code duplication that my compiler is warning me of but my tests are running.