If you are using cudnn, then different kernels might be picked due to the difference in workload (models) and GPU architectures.
You could use torch.backend.cudnn.benchmark = True to let cudnn profile the kernels and pick the fastest one. Note that the first iteration for each new input shape will be slow, since the profiling will be executed.