Hi Michael!
What you describe is a standard approach.  You should be able to
make it work, assuming you have enough good training data, and
your network is appropriate for the task.
I have some comments in line, below.



 Michael_Lempart:

My first model was trained by transforming the segmentation map into a one-hot encoded vector


Because you can transform your segmentation map into one-hot
encoding, it sounds like you are working on a single-label, multi-class
classification (segmentation) problem.


and applying a softmax after the final layer.


Applying softmax() also suggests that you have a single-label
problem.  (In such case, however, you would use CrossEntropyLoss
without the softmax() as CrossEntropyLoss has, in effect, the
softmax() built in.  You would also not use one-hot labels with
CrossEntropyLoss.)


I have noticed that labels can overlap in certain cases.


“Overlapping” labels sounds like a multi-label, multi-class problem.
I assume that you mean you sometimes have overlapping target
labels – your ground-truth annotations that you use to train – rather
than overlapping predicted labels.
But if you do have overlapping labels, how can you convert them
to one-hot encoding?


My idea was therefore to stack the 3 different segmentation maps into 3 seperate image channels, where each segmentation is binary (background = 0, foreground = 1).


This is appropriate for multi-label image segmentation.


To train the model I used the BCEwithLogits loss


This is also appropriate.  But, just to be sure, you should be feeding
the output of your final Linear layer as your logit predictions to
your BCEWithLogitsLoss  loss function, without passing it through
softmax() or sigmoid().


and during training the result looks good.


I assume that this means that your training-set loss goes down and
your training-set accuracy goes up a sensible way.


Nevertheless, when using the model to predict new segmentations for a given CT image, the model outputs the same segmentation in all three channels and I can’t figure out whats wrong…


This, of course, is not what you want.  Also, this should show up
quantitatively in that your test-set loss and accuracy should be
much worse that that for your training set.
One thing to check for:  Do you have an unbalanced dataset where
some classes (including your background class) appear much more
frequently than others?  If so, consider using BCEWithLogitsLoss's
pos_weight argument to help account for this.
Good luck.
K. Frank