Solved by ptrblck in post #2 
              
              
                If you are using cudnn, then different kernels might be picked due to the difference in workload (models) and GPU architectures. 
You could use torch.backend.cudnn.benchmark = True to let cudnn profile the kernels and pick the fastest one. Note that the first iteration for each new input shape will …