People often measure cuda times incorrectly (in this case it is even more likely as loss.item() is often a cuda synchronization point). In practice, loss calculation times should be too insignificant to be concerned. Same with memory.