Yes, I did. The reported stats are the ones after warmup phase. I am using RTX 3060.
I also discovered that I get a marginal (roughly 25% speedup) with amp using
torch.backends.cudnn.benchmark = True
Just curious on why this is the case.