Doing my mea culpa hopefully so it can help someone else at some point:
The wall clock time was measured inside a profiler context. It’s the profiler context that was slowing down the overall inference and not the PyTorch version itself (PyTorch 1.2 profiler is faster than PyTorch 1.7 profiler somehow). Needless to say measuring wall clock time with the profiler active is not desirable anyway.
TLDR: No performance regression as far as I can tell, just don’t add the profiler overhead to your measurements.