Oh I see, that’s maybe related to the fact that CUDA uses different algorithms for convolution. In fact, the algorithm is determined automatically during runtime, and sometimes the choice is not ideal. (Also, CUDA does some internal optimization, so for the smaller convolutions, it can maybe not decide which convolution algorithm should be used and it is switching back and forth and is re-estimating and re-optimizing).
Have you set the deterministic flag for the benchmarking? I.e.,
if torch.cuda.is_available():
    torch.backends.cudnn.deterministic = True