It makes not sense at all. Some samples may be harder than other, thus, their loss will be harder.
Dummy example:
Classify this:

image.jpg940×960 212 KB

Now classify this:

image.jpg600×800 109 KB

Lastly classify this:

image.jpg640×592 76.6 KB

Last case is harder, loss will be higher but you aren’t teaching the network to discern that as you aren’t updating weights.