Hi Carl,
thanks for the nice script to reproduce this issue.
You are seeing the time difference most likely due to denormal values (float values close to zero), which are apparently slower to process.
If you set torch.set_flush_denormal(True), you’ll see that both checkpoints are faster to process and have approx. the same time now.