Could you check the memory usage on your RPi and check, if you might be running out of memory while compiling on the device?
If that’s the case, you might need to add a larger swap and rerun the build.
The last time I’ve compiled PyTorch on my RPi ver1, I needed to increase the swap by quite a bit and I think it took approx. a day to finish the compilation on the device (it was the first RPi with a single core).