ZhengdiYu:

From what you told me, could I assume that torch.cuda.synchronize() could help me get absolute time of these operations ?


Synchronizations will make sure the timers grab the runtime of all operations between the sync points.
Otherwise your profiling would run into the “ghost” operations you’ve mentioned, i.e. you would most of the time profile the kernel launch only and would accumulate the complete runtime of all scheduled kernels into the next blocking operation.



 ZhengdiYu:

Why is there the difference ?


The first operation will return the max values in dim=1, so the output would contain the values and indices in the shape [b], while the second call treats the inputs tensor as a flat tensor and will only return the max. value as a scalar.
That being said, I cannot reproduce the large timing difference using this code:
x = torch.randn(1024, 1024*128, device='cuda')

nb_iters = 1000

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = torch.max(x, dim=1)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
> 0.012184044361114503

torch.cuda.synchronize()
t0 = time.time()
for _ in range(nb_iters):
    out = torch.max(x)
torch.cuda.synchronize()
t1 = time.time()
print((t1 - t0)/nb_iters)
> 0.012206459522247314