Indeed, I wonder about three things:

how much slower?
what GPU are you using?
How did you install pytorch? (conda / pip / what command?)

There are minor differences in speed between the CuBLAS in CUDA8 (which was default binary for pytorch 0.3.1) and CuBLAS in CUDA9 (default binary for pytorch 0.4.1).