You might try to follow the route android takes, providing a CMAKE_TOOLCHAIN_FILE and you might need a host protobuf, see scripts/build_android.sh for inspiration.
I will admit that I did not try but build natively on aarch64 for the time being…
Best regards
Thomas