savvas17:

Are these operations O(n)


Yes.


or does Pytorch perhaps keep track of them behind the scenes for quick retrieval?


No.

You want warmup, probably.
You don’t want to use time.time for timing. Use time.perf_counter. Or use Jupyter/IPython and %timeit
One thing to keep in mind with GPU benchmarking is to do the syncing before looking at the wall clock (before start and before end).

Best regards
Thomas