Well, so for the packages there, I used a 4GB Raspberry. Crucially, I did not set up swap. (For good reason!)
I just ran python3 setup.py bdist_wheel as usual. This builds on 4 cores. When compiling the extremely large autogenerated files (the ones wrapping ATen into autograd and Python), it runs out of memory and gets killed. Then I ran cmake manually for a bit with -j1 (1 process at a time) and when I got bored switched to -j4 again. Maybe with the 8GB I can just run -j2 throughout or something.
I’ll look at building with the Raspberry OS (64 bit Raspbian wasn’t yet available a year ago). Thank you for handing me an excuse to get the 8GB one.  It is scheduled to arrive tomorrow, so we’ll see if I have packages over the weekend.
Best regards
Thomas