Thank you for your response. I noticed that my Dataset is heavily imbalanced, there are way more pixels representing the backgrounds than the organ (label I am interested in). I have 18000 images of which only 600 have the organ class I am looking for. Is it possible that this is the reason why only one of the classes is learned?
Thanks
Nishanth