Solved by JieLei in post #8 
              
              
                After using 1.4.0.dev20191109 with Python3.7 and CUDA10.1. I get a similar performance. It seems like a PyTorch1.3.1 or CUDA 10.0 problem.