Hi Philipp!



 Philipp_Singer:

Now my first batch has weights [1,1] and my second batch has weights [2,2], and for all 4 samples the loss is 1, then both batches have the exact same overall loss. This does not sound intuitive to me as I want to weigh the second batch higher.


First, note that in practice you can use reduction ='sum', and then
sum the the two per-batch losses together to get the relative weighting
you want.  (You can divide by total number of samples, if you choose.)
You do, however, raise a legitimate point:  Splitting a batch in two, and
averaging together the weighted averages of the two sub-batches
won’t give the same result as taking the weighted average over the
unsplit batch.  But it’s a trade-off, and you can’t have everything.  I’m
willing to give this up in order to have the weighted average of a bunch
of identical losses be equal to that shared loss value.
(Note, if you take an appropriately weighted average of the sub-batch
weighted averages, you will get what you want, although, in practice,
going this route likely confuses the issue.)
Best.
K. Frank