Based on your description you might profile the CUDA init in the first iteration using the Python timer or the “wake up cycle” in case the GPU was idle before, so I would recommend to use torch.utils.benchmark to profile specific ops, as it’ll take care of the warmup iterations as well as necessary synchronizations.