Which line of code is raising the error?
I guess that some internal torchaudio.transforms might create CPUTensors and might thus yield the device mismatch error.
We could take a look at it, if you could post the complete stack trace.