Solved by my3bikaht in post #7 
              
              
                Yup, I missed that CUDA time. What if you enable optimal cudnn algorithm selection by setting torch.backends.cudnn.benchmark = True ?