OK finally got it working seems to be some incompatibility in cudatoolkit, cudnn and pytorch. In the end I got it working by compiling pytorch from source. I went for Cuda toolkit 10.0 cudnn 7.5 and pytorch master (latest greatest yay!)