Hello Peppermint!



 ppmt:

reduction = ‘mean’, no weights: 0.7963
reduction = ‘sum’, no weights: 19.11
reduction = ‘none’, no weights, manually summing losses: 19.11
I would expect that the averages loss times the batch size equals the summed loss. Here, 24*0.7963 equals 18.96 which is not identical but close to the summed loss.


Actually, everything looks good here.  According to my calculator,
24* 0.7963 = 19.1112.  (As an aside, 24 * 0.79 = 18.96.)


reduction = ‘mean’, weights in loss fct: 0.658
reduction = ‘sum’, weights in loss fct: 51.48
reduction = ‘none’, weights in loss fct, manually summing losses: 51.48
Now 0.658 * 24 (=15.79) is totally different from 51.48. Is this the intended behavior?


Yes.


How does Pytorch compute the average?


CrossEntropyLoss computes a weighted average using your weights.
In your example, you have 23 weights of value 1, and 1 weight of value
55.  So the sum of your weights is 23 * 1 + 1 * 55 = 78.  So the
denominator or the weighted average is 78 (not 24).
According to my calculator 78 * 0.658 = 51.324 (which is good
enough for ranch work).
Here is pytorch 0.3.0 script that illustrates the unweighted and weighted
CrossEntropyLoss “mean”:
import torch
torch.__version__

torch.manual_seed (2020)

pred = torch.autograd.Variable (torch.randn ((24, 2)))
targ = torch.autograd.Variable (torch.bernoulli (0.11 * torch.ones(24)).long())
targ.sum()

torch.nn.CrossEntropyLoss (weight = None, size_average =  True, reduce =  True) (pred, targ) * len (targ)
torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce =  True) (pred, targ)
torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce = False) (pred, targ).sum()

wt = torch.FloatTensor ([1.0, 55.0])

torch.nn.CrossEntropyLoss (weight = wt, size_average =  True, reduce =  True) (pred, targ) * wt[targ.data].sum()
torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce =  True) (pred, targ)
torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce = False) (pred, targ).sum()

And here is the output:
>>> import torch
>>> torch.__version__
'0.3.0b0+591e73e'
>>>
>>> torch.manual_seed (2020)
<torch._C.Generator object at 0x000001E4CA336630>
>>>
>>> pred = torch.autograd.Variable (torch.randn ((24, 2)))
>>> targ = torch.autograd.Variable (torch.bernoulli (0.11 * torch.ones(24)).long())
>>> targ.sum()
Variable containing:
 3
[torch.LongTensor of size 1]

>>>
>>> torch.nn.CrossEntropyLoss (weight = None, size_average =  True, reduce =  True) (pred, targ) * len (targ)
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

>>> torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce =  True) (pred, targ)
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

>>> torch.nn.CrossEntropyLoss (weight = None, size_average = False, reduce = False) (pred, targ).sum()
Variable containing:
 22.5843
[torch.FloatTensor of size 1]

>>>
>>> wt = torch.FloatTensor ([1.0, 55.0])
>>>
>>> torch.nn.CrossEntropyLoss (weight = wt, size_average =  True, reduce =  True) (pred, targ) * wt[targ.data].sum()
Variable containing:
 302.9537
[torch.FloatTensor of size 1]

>>> torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce =  True) (pred, targ)
Variable containing:
 302.9537
[torch.FloatTensor of size 1]

>>> torch.nn.CrossEntropyLoss (weight = wt, size_average = False, reduce = False) (pred, targ).sum()
Variable containing:
 302.9537
[torch.FloatTensor of size 1]

As you can see the results of the various “reductions” all tie out (when
the average is understood as a weighted average).


Is there any difference in training behavior when using the sum instead of the mean reduction?


Not a lot.  In the unweighted case, the loss using “sum” will be nBatch
times bigger (24, in your example) than with “mean.”  In effect, this
means that you would be using a learning rate 24 time bigger.
Learning rate does matter, so you would probably want to use a
proportionately smaller learning rate if you use “sum” rather than
“mean.”
In the weighted case, the ratio between “sum” and “mean” will depend
on the number of “1” target values in your batch (as illustrated by the
above script).  So your effective learning rate would vary from batch
to batch based on the number of "1"s.  But I think this batch-to-batch
variation would largely average away over multiple batches.
One last note:  Because you’re training a binary classifier, you would
probably be modestly better off using BCEWithLogitsLoss 4 and its
pos_weight argument.  (To do this you would change the last layer
of you network to emit only a single prediction (per sample in the
batch), rather than two.)
Good luck!
K. Frank