Your general approach is very good.
Are you looking at the first iteration or are these profiles from later steps?
In the former case, you should skip the startup and first iterations, as e.g. the caching allocator would need to allocate memory, which can later be reused.
You can also add markers using:
torch.cuda.nvtx.range_push('my name')
[...]
torch.cuda.nvtx.range_push('my nested name')
[...]
torch.cuda.nvtx.range_pop()
torch.cuda.nvtx.range_pop()

to get a better idea which method is using which kernels.