Since you are compiling from source, please make sure you have the latest version (of pytorch and torchaudio) from master. This may have been fixed by https://github.com/pytorch/audio/pull/326 18.
Note to self: This should not be related to https://github.com/pytorch/audio/pull/339 11.