As far as I know, there isn’t per se and I for one compile PyTorch on my RPi4, even if it takes quite a while. You could probably use distcc to offload the compilation to somewhere faster.
That said, the Android compilation works as a cross-compilation, so libtorch can be cross-compiled and it probably is a question of adapting some cmake bits to do the right thing.
Best regards
Thomas