You could try to use --no-deps while installing torchaudio, but note that pip or conda would only downgrade PyTorch, if torchaudio specifies a specific PyTorch version as a dependency, so your installation might break in case your source build is not compatible with the desired torchaudio version.
A proper approach would be to either build torchaudio also from source or to try to install the nightly releases, which should have relaxed requirements.