Hi apytorch!



 apytorch:

However, I wonder if it’s just overthinking the issue,


I think that you are just overthinking it.
Having said that, my intuition on the matter – without any evidence
to back it up – is:


Class weights in the loss function are rough values to help
things along a little bit.  Your training process and results
should be robust with respect to the exact values of the class
weights you use.


My feeling is that estimating your class weights from the entire
training set, and then using the same weights for all batches,
should be sufficient, and probably even preferable.  It is true that
(unless you construct your batches so the the number of samples
from each class is the same from batch to batch) your class
distributions will vary from batch to batch.  But because your
learning rate is “small,” your training process will, in effect,
average over several batches, smoothing out the varying class
distributions.  This will be even more true if you use momentum.


Best.
K. Frank