Hi all. I am a graduate student working on a project about Model inference time optimization.
I am using this very simple code snippet to benchmark the inference time of resenet50:
    for i, (batch_input, batch_target) in enumerate(data_loader):
        batch_input_var = batch_input.to(device)
        torch.cuda.synchronize()


        # time.sleep(0.2)
        start = torch.cuda.Event(enable_timing=True)
        end = torch.cuda.Event(enable_timing=True)
        start.record()
        with torch.no_grad():
            output = net(batch_input_var)
        end.record()
        torch.cuda.synchronize()
        print(start.elapsed_time(end))

It gives me 10.5ms consistently.
However, if I uncomment time.sleep(0.2), or do something else which takes around 0.05s to 0.2s, the results start to be highly inconsistent, ranging from 12.7ms to 26.1ms. If I set the sleep time to say 2 seconds, the results become consistent again, around 15ms.
Please I am wondering if you have any thoughts on this?
Thanks!