ZhengdiYu:

Anyway, the problem has gone even though I haven’t figured out why it is that way.


If you don’t synchronize the code before starting and stopping the timer, you won’t capture the real runtime of the CUDA operations, since they are executed asynchronously in the background.
Basically your Python script will call the CUDA operation, which will be scheduled on the GPU and executed, while the Python script can just execute the next lines of code.
E.g. in this code snippet:
t0 = time.time()
y = x.sum() # x is a CUDATensor
t1 = time.time()

you would only profile the Python overhead of calling the summation CUDA kernel, not the execution itself.
The code would block automatically, e.g. if you are trying to use a result tensor:
# async execution
output = model(x)
loss = criterion(output, target)
loss.backward()
# blocking
print(loss.cpu().item())

These synchronization points might be in different parts of your code, so that your profilng result would accumulate all previous CUDA operation into the first blocking operation, which will yield wrong results.