I have been using pytorch 0.3 with CUDA 9.1 for a few weeks now on ubuntu 16.04 and non-anaconda - just using the tensorflow-gpu-py3 docker image with nvidia-docker and it works fine.
Also this is a non-anaconda install - just used pip and the cu90 installer.