I did not look in details but the repo you linked contains a full cuda implementation of the code and the readme contains a discussion of python vs c++ code here 6.
For cuda, the whole API is asynchronous, so at any point in you python code, there might be stuff running from the previous lines. To get proper timings you should always call in the following order (not that adding many cuda sync can slow down you process):

torch.cuda.synchronize()
current_time = time.time()