Hi all,
from my understanding the weight parameter in CrossEntropyLoss is behaving different for mean reduction and other reductions. I believe in case of non-mean reductions the sample loss is just scaled by respective class weight for that sample. Yet, in the case of mean reduction, the loss is first scaled per sample, and then the sum is normalized by the sum of weights within the batch.
The important thing to me is the last part, namely that it is normalized by weights within the batch. Is this observation of mine correct? I wonder what is the idea behind that? If I am using weights to e.g., balance across the dataset, wouldn’t I want to normalize globally?