I just installed it via pip on Google Colab directly from the GitHub repo and it works fine.
pip install git+git://github.com/pytorch/audio


Screen Shot 2018-05-22 at 1.57.46 PM.png880×193 26 KB

Note that you need to install the required dependencies via apt-get first as mentioned in the GitHub repo. Also you need to have PyTorch installed, otherwise you will get an error during the install.
EDIT:
The dependencies I mentioned above are the ones listed for Ubuntu Linux (https://github.com/pytorch/audio#dependencies 33). Worked fine for me in the Colaboratory Notebook:
apt-get install sox libsox-dev libsox-fmt-all