The reported times are likely inaccurate because you are not taking into account asynchronous cuda execution, so synchronizing .item() call appears as if it’s taking a long time, but in fact it is reporting the time taken by previous asynchronous calls
in pytorch 1.5 computation of clip_grad_norm was changed to avoid synchronizing .item() calls.