So this has two parts:

Build libtorch on aarch64,
build libtorch on Android.

Which one are you most interested in?
For 1) you can build PyTorch natively on the largest Raspberry Pi 4 or NVidia Jetsons pretty much by running setup.py. The caveat is that the build may (will) crash with out of memory if you run with -j4. My solution to that has been to then build the large C++ with -j1 and afterwards continue with -j4. More fancy things like distcc probably get better results yet.
For 2) check out the Pytorch mobile build instructions 66.
Best regards
Thomas