I built and used TorchAudio on Raspberry Pi OS (wheels are linked from the page linked above in the January 2021 update).
The main caveat is that it uses Python 3.7. If you need a newer Python version, you’d need to recompile, but the instructions should be broadly the same.