Hi Stark!
Okay, now I think I understand your use case.
Unfortunately, I don’t think I have anything useful to say about “instance
segmentation.”  Perhaps some experts will chime in.
I do stand by my technical comments about the immediate causes of the
errors your reported.  Some clarifying comment appear in line, below.



 stark:

The problem is I cannot figure out a suitable encoding scheme that can work with the loss functions implemented in the PyTorch.


I’m not aware of any built-in pytorch loss functions that are directly
applicable to instance segmentation, but, like I said, I’m not knowledgeable
about this use case.


Overall, I followed this sequence:
RGB images + 5D labels (input) —> model —> 5D logit output —> sigmoid() —> BCELoss()/CrossEntropyLoss()


When you say “5D labels (input)” the only thing I think you can be referring
to are the ground-truth labels (“target”) that you use for training.  Do you
really feed in the ground-truth labels as input into your model?
Also, you can’t use the output of the same model as input to both
BCELoss and CrossEntropyLoss (if they are using the same target).
Wrong shapes.





 KFrank:

if your input is of shape [nBatch, nClass, width, height] , your target should have shape [nBatch, nClass, width, height] . It appears that your input and target don’t satisfy this relationship.


The number of input channels to a model can be different from the number of output channels. Please correct me if I’m wrong. I’ve seen so many segmentation models designed this way while using the CrossEntropyLoss().


As I mentioned in my previous post in the context of BCELoss, I am using
input to refer to what pytorch calls the input to your loss function.  This
is the output of your model, not the input to your model.  (This bit of pytorch
terminology can lead to confusion.)
Note, in the bit you quoted, I gave the wrong shape for the target fed to
CrossEntropyLoss:  It should be [nBatch, width, height], without an
nClass dimension.





 KFrank:

Are your really applying your model to RGB images?


Yes, my model takes a batch of [3, 128, 128] images and [5, 128, 128] of target labels as input.


Again, does your model really take target labels as input.  Or are you
referring to your overall training procedure?
Leaving this aside, if I understand “target labels” to be the target
you feed to BCELoss, then your shapes do makes sense.  To use my
preferred terminology, the input to your model is a tensor of shape
[nBatch, nChannel = nColor = 3, 128, 128].  (And, yes, it is
common to refer to the three colors as “channels.”)
Your model then outputs predictions (the input to your loss function) of
shape [nBatch, nClass = 5, 128, 128].  This would be appropriate for
BCELoss and (something analogous to) a multi-label, multi-class problem.
The target you feed your loss function should, for BCELoss, have the
same shape ([nBatch, 5, 128, 128]) (which it appears it does), while,
for CrossEntropyLoss, it should have a shape lacking the nClass
dimension ([nBatch, 128, 128]).


By the way, assigning 1 to every foreground pixels in target labels (as in the figure below) let BCELoss() to produce reasonable values.


Yes. By doing so you are (properly) training a conventional multi-label,
multi-class classifier.


However, I’m not sure how the multiple occurrences of foreground objects from the same class can be identified in the model output (since the model cannot infer this from the labels).


But, also yes, doing so won’t perform instance segmentation for you
(which, again, I don’t know how to do).  That is, I agree with your analysis
that, while setting the all of the foreground pixel values to 1 does fix your
problem of bad BCELoss values, doing so throws away the information
you need to perform instance segmentation.
Best.
K. Frank