Class imbalance in image segmentation is a notoriously difficult problem. I got decent results with 2 things :

using an architecture with multiscale representation, like HrNetv2
using the loss suggested in this paper : https://arxiv.org/abs/1708.02002 21, the idea is that the loss isn’t decreased much for confident correctly classified pixels. This way, most of the parameters updates are concentrated on classes that are difficult to classify.