Hi,
I would say this is due to cudnn internals.

If you disable cudnn, you should use only our implementation and should not see this jump
cudnn has it’s custom algorithms depending on the input size
You can enable cudnn benchmark (torch.backends.cudnn.benchmark=True) and see if you see better behavior, it should pick the best algorithm and so remove such artifacts.