Hi Alisson!



 Alef93xx:


Target formatting is really my biggest issue here, as you said. I’ve seen in other threads that we must convert RGB colors to class indices, but I have no idea how to do that. I’ve even tried it, but nothing has worked yet.



Let me make a number of comments before outlining how one
might convert RGB colors to integer class labels,
First, just to use a clear-cut word, let me call your ground-truth,
annotated (labeled) images “masks.”  So your dataset consists
of a bunch of images that will be input to your model (for both
training and validation) each with a corresponding mask that is
used either to train your model or evaluate how well your trained
model is performing its segmentation task.
First, are your training and validation datasets in the same format?
Do your training and validation datasets come from one larger
combined dataset that was split (perhaps randomly) into the two
separate training and validation datasets?
What do your input images look like?  Are they color images?
Grayscale?  Do they look like sensible images that appear to the
eye to have structure that could be segmented?
What do your masks look like?  Do they have blobs of different
colors in them that line up sensibly with the segments in the
corresponding input images?
Before processing, it sounds like your masks  are some sort of color
images.  What format are they in?  Do they have three separate RGB
channels?  Are they 32-bit RGB images (that might only be using
24 bits under the hood)?  Are they 8-bit palettized images?
What format are your masks in after any processing that may occur
in your dataloader?
I would suggest that you print out the shapes of your input-image
and mask batches both immediately after they are delivered by
the dataloader, and then again immediately before they are fed
to your model and loss function.  (The reason to print out the
shapes at these two different locations in your code is to verify
that your squeeze()s and permute()s are doing what you want.)
In the typical case (It doesn’t have to be this way, but it would be
typical.), a batch of input images would have shape
[nBatch, nChannel, height, width].  nChannel would typically
1 for grayscale images and 3 for color images.  (nChannel could be
3 for grayscale images that happen to be formatted as color images.)
Note, if you are passing color images to your model, it will only work
if your input images (by the time they get passed to your model) are
multi-channel (almost certainly three-channel RGB) images.  Your
model won’t work with 32-bit color nor 8-bit palettized color images.
The output of your model will typically have shape
[nBatch, nClass, height, width].  You should also print out
the shape of your model output images to verify this.
When used for training with CrossEntropyLoss your masks must
have shape [nBatch, height, width] (with no nClass nor nChannel
dimension), and the values of the mask pixels must be integer class
labels that run over 0 to nClass - 1.  This is what CrossEntropyLoss
expects.
(It would be possible to have masks of a different shape or format
for your accuracy calculation – you don’t have to follow the
requirements of CrossEntropyLoss, but there is likely no benefit
to using a different format, and doing so would only confuse the issue.)
I would recommend making a simplified “test-bed” version of your
code – get rid of as much extraneous stuff as you can.  Get rid of
the checkpoint stuff, cuda, amp, scaler, etc.  I would keep the transforms
as they could modify the shape / format of the images and masks
that are being passed to your model, loss function, and accuracy
calculation, and that’s part of what you need to check.
Then try passing a single batch of train_loader images through
your model and loss function (printing out the shapes, as discussed
above) and try passing a single val_loader batch of images through
your model and accuracy calculation (again printing out shapes).
Make sure that everything lines up the way it should, and if it doesn’t
make sure you understand the details of what isn’t right, and what
you need to do, conceptually, to fix it.
Now to the question creating masks that consist of integer class
labels.
I will assume that your masks start out being some sort of color
images where each of a discrete set of colors (in your case,
presumably nClass = 21 different colors) indicates a specific
class.
Converting your masks to consist of integer class labels could be
done in your dataloader or in a separate processing step after
your dataloader.  I would probably put this processing step in the
dataloader, but it doesn’t really matter.
You need to write the code that reads through all (or at least a
representative sample) of your masks and makes a table of all
the colors that show up in the pixels.  (Conceptually, it doesn’t
matter whether the colors are encoded with three RGB channels
or as 32-bit or 8-bit colors – you just need to keep track of the
distinct colors that show up.)  Each distinct color corresponds to
a class.  Because you say that you have 21 classes, you had better
have no more that 21 distinct colors.  For example, if you discover
that your training masks have 25 distinct colors, then your masks
are telling you that you have 25 classes, not 21, so you would have
to sort that out.
Now make a dictionary (consider it to be a look-up table) whose
keys are the 21 distinct colors and values are the integers 0
through 20.  The assignment of integer class label to mask color
can be arbitrary.  If your classes have some “meaning,” you are
welcome to assign integer class labels that somehow reflect that
meaning for your own convenience, but the choice won’t affect
how the model training progresses.  (The class-label assignment
does have to be consistent across your dataset – both your training
and validation data jointly.)
(As an aside, for this kind of semantic segmentation, we often think
of an image being segmented as having 'foreground" objects that
form the classes and an “uninteresting” background.  For training,
however, the background pixels still form a class, and are treated
no differently than the “foreground” pixels.  For your convenience
(It doesn’t affect the training.), you might choose to use 0 as the
integer class label for the background pixels.)
Now for each mask read through the pixels in the mask tensor and
create a new mask tensor of dtype = torch.int64 (the same as
torch.long) and typically of shape [height, width].  As you write
your new mask tensor on a pixel-by-pixel basis, translate the
corresponding color-valued pixel from your original mask into the
proper integer class label by passing it through the look-up table
you built.
Good luck.
K. Frank