Yes, this would give you the accurate timings. Once you have the profiles, you should remove it again to allow the asynchronous calls again.
Note that the first iterations (or each iterations using a new input shape) will be slower, if you are using torch.backends.cudnn.benchmark = True, as cudnn will run some benchmarking on the current workload.
Also, usually you would add some warmup iterations, but this might not be necessary if you just want to narrow down the bottleneck.