Because your losses are most likely in a different range (sum vs. mean), you would have to change your learning rate accordingly to get the same weight updates.
In your first example your learning rate might just be too high for the high loss values.