Thanks @ptrblck for the explanation.



 ptrblck:

The background class would be necessary, if your model is supposed to predict the class of each pixel.
During training you could ignore the background class pixels, but then your model won’t be able to predict the background during evaluation and the predicted segmentation would contain only the two valid classes.


As per reporting the model performance both at training and validation phase, the background class is not needed. However, I am of the idea that it might help when trying to visualized the prediction of the model.
Moreso, in computing the background, I stacked the annotated (masked) label of the given classes and consider its complement i.e having 1’s for 0’s of stacked label and vice versa with little attention paid to overlapping pixel region.



 ptrblck:

I’m unsure if weighted sampling would work nicely for segmentation use cases, since each sample would most likely contain more than a single class. Drawing a particular sample would thus increase the class count of multiple classes and you would end up with the limitations for a multi-label classification use case.


I appreciate your clear explanation. In this case, how will one handling class imbalance in segmentation cases - in particular get the class weights for each class rather than using robust network like HighResNet as someone suggested in one of the forum?