Hi Rory!



 roryw10:

Does that mean i need to know across all my images how many pixels are 1 v 0  If so is there any best practice on how to do this?


There is nothing magic about the exact choice of pos_weight.  It
provides a rough reweighting of positive samples vs. negative.
Nothing goes wrong if you choose one reasonable value instead
of another.
I don’t think that there is any benefit in trying to choose pos_weight
on a per-batch basis – just get an approximate estimate of the ratio
of “1” pixels to “0” pixels in your entire training set.  There would be
nothing wrong with counting the exact numbers of "1"s and "0"s in
your training set, but if you want to save time, you could just count
the pixels in a representative sample of your images.  You could
even count a representative sample of pixels within an image (in
your representative sample of images).  You just need to count
enough (representative) pixels that you get a reasonable statistical
estimate of the number of "1"s vs. "0"s.
The reasonable and simple choice for pos_weight is then
number_of_0s / number_of_1s.  (The idea is that if your training
set consists of almost all "0"s, then your classifier will do a very
good job by always predicting “0” – clearly not what you want.
The above choice of pos_weight means that in aggregate your
“0”-pixels and “1”-pixels will now contribute to your loss function
– and hence your training – with approximately equal weight.)
(A side note on a potentially confusing bit of terminology.  The “number
of classes” for the length of the pos_weight vector in your use case
is 1.  The one class is, let’s call it, “foreground”, and a pixel can be
either the binary choice “yes foreground” or “no foreground”.  Your
network will provide one logit output per pixel.  If we were
speaking in terms of a multi-class classifier (using, for example,
CrossEntropyLoss) we would say you had two classes,
“yes foreground” and “no foreground.”  The reason we don’t
stamp out this confusing terminology for BCEWithLogitsLoss
is that you can use BCEWithLogitsLoss for a multi-label,
multi-class classification problem where you can have n classes,
any number of which can be “yes” or “no” at the same time.  This
is then a set of n binary classification problems, but we (smartly)
train a single classifier network that performs the n binary
classifications at the same time, (usefully) sharing features and
training, etc.  In this case we say we have n classes, not 2^n
classes – that is, we don’t count the number of combinations of
"yes"s and "no"s as the number of classes.)
Good luck!
K. Frank