Thank you so much for your help. Some update about the training process, after 100 epoch the accuracy is about 80% so I think model is fine.



 KFrank:

Is it possible that one of your classes is a “background” class and
a large fraction of your pixels are background pixels?


Yes, the labels contains large amount of background. Thank you for your suggestion about the CrossEntropy with weight. I think it will be useful and I am trying it out.