Let me apologize for digging this topic, but there is one thing I want to make sure I understand correctly. Let’s say I have a training loop that performs both CPU bound and GPU bound operations in a sequence. It:

Loads a batch of data from the disc
Converts the data to PyTorch Tensors on a selected device
Do the necessary calculations(forward, loss, backward)

The script runs slower then I expected it to run and I wanted to time the respective blocks to see what is the contribution of each of this major actions on the time of processing one batch. My first approach was to try with the python’s time module, but it fooled me and I ended up optimizing a function that should take just a fraction of the total execution time for a whole day, and then when I used the method described in @richard answer, it turned out that this small function really is meaningless when it comes to the total execution time. So, my question is now, is the above described method device agnostic?  Can I also use the cuda events to time the parts of code that are fully CPU bound, like loading the data from disk? Will the time printed by the below snippet give me an accurate result?
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
some CPU bound operations, i.e. loading data...
end.record()

# Waits for everything to finish running
torch.cuda.synchronize()

print(start.elapsed_time(end))

I would be grateful for clearing this out for me, I just want to avoid misunderstanding.