How are you profiling these operations? If you are using the GPU, remember that you would need to synchronize the code to get valid profiles, as CUDA operations are executed asynchronously.
If you’ve already taken care of it, I guess the backward operation in loss2 adds the overhead.