There is something called dice loss (which is quite similar to F1-score but can also be soft to be differentiable) which is used regularly in semantic segmentation tasks. The dice score (or F1-score) can then also be used as a metric to evaluate one’s model on the validation set.