(This was originally a question but I found a solution)
Those steps worked for me:

If your compute server has a CUDA module that you need to load using module load, DON’T load it before building pytorch (this may not be needed)
Set $CUDA_HOME to where cuda is installed. (for me it was: /software-gpu/cuda/8.0.44)
Install torchvision BEFORE building pytorch, as with conda torchvision installs cudatoolkit and this screws up the CUDA paths and cause an error when doing import torch

Run python setup.py clean before python setup.py install