Thanks I can reproduce the different timing results using time.sleep.
However, if I add a real workload, e.g.:
def fun2():
    for _ in range(200):
        np.matmul(np.random.randn(1024, 1024), np.random.randn(1024, 1024))

if __name__ == '__main__':
    device=0
    ...
    for i, (batch_input, batch_target) in enumerate(data_loader):
        batch_input_var = batch_input.to(device)
        torch.cuda.synchronize()

        #time.sleep(0.2)
        fun2()
        ...

I get valid results again (although the code overall will have a large delay), so I assume time might distort the profiling somehow.
I’ll ask around, since this is new to me.