ptrblck:

torch.cuda.synchronize()


Thanks. But the results still different. Let do a simple check. In the code, I will use same conv2d operator as
torch.cuda.synchronize()
with torch.autograd.profiler.profile(use_cuda=True) as prof:
    out1 = net1(input)
    out2 = net1(input)
    #out.backward()
print(prof)

The result must be same but the first conv spends 2411us, while the second one spends 43us. It is same conv2d but a lot of difference
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------
Name                          CPU time        CUDA time            Calls        CPU total       CUDA total
---------------------  ---------------  ---------------  ---------------  ---------------  ---------------
conv2d                      3116.058us       2411.520us                1       3116.058us       2411.520us
convolution                 3100.801us       2406.400us                1       3100.801us       2406.400us
_convolution                3089.441us       2401.280us                1       3089.441us       2401.280us
contiguous                     5.759us          3.072us                1          5.759us          3.072us
empty                          3.426us          3.072us                1          3.426us          3.072us
cudnn_convolution           3053.347us       2385.952us                1       3053.347us       2385.952us
conv2d                        58.446us         43.008us                1         58.446us         43.008us
convolution                   55.017us         37.888us                1         55.017us         37.888us
_convolution                  51.780us         31.744us                1         51.780us         31.744us
contiguous                     1.626us          2.048us                1          1.626us          2.048us
empty                          2.208us          2.048us                1          2.208us          2.048us
cudnn_convolution             39.026us         16.384us                1         39.026us         16.384us