I published ARM64 binaries 51 of PyTorch compiled on the Raspberry  (I actually compiled 1.4, too, just didn’t upload yet). You would need a 64bit distribution (eg Debian for the Raspberry Pi 3) or a 64 bit kernel from the Raspberry Inc and arm64 chroot.
There also are ARM32 binaries 53 from @LeviViana . At least in September, some things like JIT tracing didn’t work when I built on ARM32, Levi would know if he fixed it or it’s still open.
Best regards
Thomas