Solved by belval in post #4 
              
              
                Doing my mea culpa hopefully so it can help someone else at some point: 
The wall clock time was measured inside a profiler context. It’s the profiler context that was slowing down the overall inference and not the PyTorch version itself (PyTorch 1.2 profiler is faster than PyTorch 1.7 profiler some…