I would try compiling pytorch 0.4 from source, in my experiments it is significantly faster. There is also a CPU optimized library called NNPACK that pytorch can use, if compiled with NNPACK support it should run faster. Finally, python 2 is known to be faster than python 3.