I am trying to train a U-Net for image segmentation. My dataset consists of 80x80 pixel images (so 6400 pixels per image), and each image can be segmented into 3 parts: primary background, secondary background, and a third class that can be any one of 9 separately defined classes. So I have 11 classes in total.
My goal is to use image segmentation to determine what that third class is for each image. I decided to sample roughly 1000 images from the dataset and this is what I observed: on average ~5587 pixels belonged to the primary background class, ~802 pixels belonged to the secondary background class, and ~11 pixels belonged to the third class (which again, can be any one of the nine special classes).
Obviously, this is a very unbalanced dataset, so I was thinking about using weighted cross-entropy loss as my criterion. How should I go about choosing these weights? Any input will be appreciated.