Hello tom. first of all, thank you for fast and detailed reply !
After i read, i tried to use time.time() and torch.cuda.synchronize() to profile the elapsed times.
However, the results are still inaccurate.
For example of https://github.com/facebookresearch/moco/issues/66 20,
i) time.time() w/o torch.cuda.synchronize()

shuffle time: 0.5993 s
inf_time: 0.1185 s

ii) use torch.cuda.Event & torch.cuda.synchronize()

shuffle time: 2.72 ms
inf_time: 59.88 ms

iii) use time.time() w/ torch.cuda.synchronize()

shuffle time: 0.0649 s
inf_time: 0.0587 s

Although i use torch.cuda.synchronize(),
the shuffle time is still over-estimated.