After some more investigation, using the release 1.5.0 android, I was able to get the app to compile but not run the PyTorch. So I got almost the same issue but with a slightly different error message:
E/SoLoader: couldn't find DSO to load: libpytorch_jni.so caused by: dlopen failed: 
cannot locate symbol "_ZN8facebook3jni11JByteBuffer5orderENS0_9alias_refINS0_10JByteOrderEEE" 
referenced by "/data/app/com.offline-EWIa3xVhdQYckXATRsmgrw==/lib/arm64/libpytorch_jni.so"... result: 0

The symbol of _ZN8facebook3jni11JByteBuffer5orderENS0_9alias_refINS0_10JByteOrderEEE differ from _ZN8facebook3jni10JByteOrder11nativeOrderEv of the original question.
When trying the proposed solution (which is to get the fbjni from PyTorch rather than RN one, it produced the same error