perhaps calling .cpu() to collect results at the end of my operations to account for the async?

You can do something like:
torch.cuda.synchronize()
start = time.time()

# your_op

torch.cuda.synchronize()
elapsed = time.time() - start


how does the development team think of overhead?

I am not working on perfs explicitly so the definitive answer will definitely be given by the relevant people on github 
But even though the Big-O notation is a good hint, I think actual runtime will be the most important.
Also such choice can depend on the input size, batch size, etc.
cc @VitalyFedyunin what do you think?