Hello Awesome!



 AwesomeLemon:

However, once we introduce weight, results change

F.nll_loss(mi, target, weight=w, reduction=‘none’).mean()
F.nll_loss(mi, target, weight=w, reduction=‘mean’)



Passing weights to NLLLoss (and CrossEntropyLoss) gives, with
reduction = 'mean', a weighted average where the sum of weighted
values is then divided by the sum of the weights.
In your `reduction = ‘none’ version:
F.nll_loss(mi, target, weight=w, reduction=‘none’).mean()

by the time you get to the call to pytorch’s tensor .mean(), the weights
are no longer available, so .mean() cannot divide by the sum of the
weights.
An example script illustrating this can be found in this post:

  
    
    
    Weight in cross entropy loss 
  
  
    Hello Mainul! 

When using CrossEntropyLoss (weight = sc) with class weights 
to perform the default reduction = 'mean', the average loss that 
is calculated is the weighted average.  That is, you should be dividing 
by the sum of the weights used for the samples, rather than by the 
number of samples. 
The following (pytorch version 0.3.0) script illustrates this: 
import torch
torch.__version__

sc = torch.FloatTensor ([0.4,0.36])
loss = torch.nn.CrossEntropyLoss (weight = sc)
input = torch.au…
  


Best.
K. Frank