I didn’t - mainly because I don’t have a Raspberry Pi - but it should be relatively straightforward if you do it on a raspberry itself. I think Soumith and others did work on compiling for ARM for NVidia Jetson.
If you need to cross-compile, maybe the cross-compilation of libtorch on android that I recently built 24 can be of inspiration. That does not build the Python module.
One thing to keep in mind is that you absolutely want NNPack when running on an ARM CPU.
PyTorch on Raspberry Pi seems a very exciting project!
Best regards
Thomas