After reinstalling pytorch multiple times through conda I could not make the script work.
I identified the problem by running torch.version.cuda in the python console on the server and realising python was using CUDA 9 instead of CUDA 10. The command torch.__file__ then showed that it was not even using torch from my conda environment, it was using another torch install on the server. Once I got rid of the other pytorch install torch.version.cuda showed CUDA 10 and everything works now.