I think I’ve made some progress, but I’m stuck on something and hope someone knows the answer.
The approach I took was taking tools/build_libtorch.py and tools/build_pytorch_libs.sh, and making changes to the cmake arguments with the help of looking at what scripts/build_android.sh was doing. So setting ANDROID_NDK, CMAKE_TOOLCHAIN_FILE and so on.
There were some errors that I managed to get around by setting -DCMAKE_CROSSCOMPILING=1, -DRUN_HAVE_STD_REGEX=0,  -DANDROID_STL="c++_static" and LDFLAGS="-llog".
But now I’m getting errors related to protobuf. I noticed a warning
WARNING: Target "libprotoc" has EXCLUDE_FROM_ALL set and will not be built by default but an install rule has been provided for it.  CMake does not define behavior for this case.
WARNING: Target "protoc" has EXCLUDE_FROM_ALL set and will not be built by default but an install rule has been provided for it.  CMake does not define behavior for this case.

And the (I think) relevant code has a comment 2 saying that pending a MR the behaviour could be changed. And the MR has been merged. But I’m not knowledgeable enough to know how to change the code… edit: found the issue 9
The errors I’m still getting are
[...]/t/pytorch-cpu/third_party/onnx/onnx/onnx_onnx_torch.pb.h:38:3: error: expected expression
  static const ::google::protobuf::internal::ParseTableField entries[];
  ^
[...]/pytorch-cpu/third_party/onnx/onnx/onnx_onnx_torch.pb.h:37:17: error: variable has incomplete type 'struct ONNX_API'
struct ONNX_API TableStruct {
                ^
[...]/pytorch-cpu/third_party/onnx/onnx/onnx_onnx_torch.pb.h:37:8: note: forward declaration of 'protobuf_onnx_2fonnx_5fonnx_5ftorch_2eproto::ONNX_API'
struct ONNX_API TableStruct {
       ^
[...]/pytorch-cpu/third_party/onnx/onnx/onnx_onnx_torch.pb.h:45:6: error: variable has incomplete type 'void'
void ONNX_API AddDescriptors();
     ^
[...]/pytorch-cpu/third_party/onnx/onnx/onnx_onnx_torch.pb.h:45:14: error: expected ';' after top level declarator
void ONNX_API AddDescriptors();
             ^
             ;