Hi Stark!
It’s not entirely clear to me what you are doing.
But to answer two technical questions:



 stark:



nn.BCELoss() : reported loss for the batches was either too low i.e. {0.0310, 0.0470, 0.1696,…} or negative i.e. {-0.2363, -0.0790, -0.1972,…}.



If I take at face value the images of numbers you posted in your image
of textual information by assuming that these are your actual target
values, then they are outside the [0.0, 1.0] range required by
BCELoss (and BCEWithLogitsLoss).  BCELoss requires its target
values to be probabilities between 0 and 1.  Hence your unreasonable
loss values.


BUT sigmoid layer had already normalized the target values in interval [0,1]. (?)


Do you really mean that you have passed your target values through
Sigmoid?  Or do you mean that your input to BCELoss (the output
of your model) has been passed through Sigmoid?




nn.CrossEntropyLoss(): This loss gave an error (as reported below).



CrossEntropyLoss requires an input and target with different shapes,
where input has an nClass dimension, and target does not.  For
example, if your input is of shape [nBatch, nClass, width, height],
your target should have shape [nBatch, nClass, width, height].
It appears that your input and target don’t satisfy this relationship. Hence
your error.
Now some comments:


The goal is to perform instance segmentation with input RGB images


Are your really applying your model to RGB images?


Logit predictions were obtained on a batch data [64, 5, 128, 128]


If you were using RGB images, I would expect your model to take a batch
of data with shape [64, nColor = 3, nChannel = 5, 128, 128] (or
something).
Are you somehow folding your RGB channels into your larger set of
five channels?


The ground truth label is multi-channel …  Each label consists of 5 non-background classes.


Does the number – 5 – of “ground truth” channels have anything to do
with the apparent number – also 5 – of input channels, or is this just a
coincidence?


there are different instances in each channel denoted by unique random numbers.


What does this labelling scheme mean?  The first image of a number in
your image of textual information is 0.  The second (distinct) number is
32.  What does 32 mean?  Is it really random?  Are you expecting your
network to learn the sequence of random numbers generated by your
random number generator?
Does a “32” in “ch #0” mean the same thing as a “32” in, say, “ch #2”?
Does a “32” in sample 3 of a batch mean the same thing as a “32” in,
say, sample 12 of that batch?
Good luck.
K. Frank