Answers.Body: <p>I agree fully with @a crazy Minon's answer.  I will just slightly expand on it and provide a couple of additional references.</p>
  <p>While Dice is a popular metric for evaluating segmentation, it is certainly not the only one.  You are right in thinking that pixel accuracy is a poor choice of evaluation metric.  The main issue is that it performs poorly when when there is class imbalance, which is often the case in imaging data.</p>
  <p>I will add that Intersection over Unions (IoU) is another metric that is frequently employed to evaluate segmentation performance.  It is also known as the Jaccard Index.  The articles <a href="https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2" rel="nofollow noreferrer">&quot;Metrics to Evaluate your Semantic Segmentation Model&quot;</a> and <a href="https://www.kaggle.com/code/yassinealouini/all-the-segmentation-metrics" rel="nofollow noreferrer">&quot;All the segmentation metrics!&quot;</a> provide good simplified introductions to various commonly used segmentation metrics.  While Dice and IoU are similar and are positively correlated, they are not equivalent, as explained by this StackOverflow <a href="https://stats.stackexchange.com/questions/273537/f1-dice-score-vs-iou/276144#276144">answer</a>.  New metrics are also being developed--such as the <a href="http://www.irisa.fr/lagadic/pdf/2017_ppniv_fernandezmoral.pdf" rel="nofollow noreferrer">Boundary Jaccard</a>--to overcome limitations of current metrics, and comparisons of these metrics have been published for specific applications (see example <a href="https://www.sciencedirect.com/science/article/pii/S0010482521002912" rel="nofollow noreferrer">ref</a>, which lists 33 evaluation metrics for segmentation in Table 1).</p>
  <p>Finally, if your interest is really for one class, then accuracy, sensitivity, and specificity for segmenting that class alone can be useful metrics.</p>
  <p>The powers that be at your institution are wise in asking for multiple evaluation metrics because each metric has its limitations and no single metric can fully capture the performance of a segmentation network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-30T20:04:59.917Z
Answers.LastActivityDate: 2022-10-31T01:14:41.407Z
Answers.LastEditDate: 2022-10-31T01:14:41.407Z
Answers.UserDisplayName: Snehal Patel
Answers.Body: <p><a href="https://metrics-reloaded.dkfz.de/" rel="nofollow noreferrer">https://metrics-reloaded.dkfz.de/</a> -&gt; Link to cite.</p>
  <p>This is an excellent site to see how to gauge which type of task and metric to use.</p>
  <p>Since you specifically want segmentation, just opt for that and have a look!</p>
  <p>Reference: <a href="https://arxiv.org/abs/2206.01653" rel="nofollow noreferrer">https://arxiv.org/abs/2206.01653</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T16:40:25.910Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-29T13:08:53.127Z
Answers.LastActivityDate: 2024-01-30T11:43:50.580Z
Answers.LastEditDate: 2024-01-30T11:43:50.580Z
Answers.UserDisplayName: Academic
Answers.Body: <p>Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for all classes together, commonly referred to as micro and macro averages.</p>
  <p>Taking it further, you may wish to have a metric more robust to changes in the threshold. Here the Area under the Curve (<a href="https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5" rel="nofollow noreferrer">AUC</a>) metric is commonly used.</p>
  <p>For a more sophisticated analysis you may also be interested in perceptual losses. These quantify how similar an image looks as perceived by people. This is particularly useful if say the shape of the prediction is important but small shifts or scaling does not matter. Have a look at <a href="https://en.wikipedia.org/wiki/Structural_similarity" rel="nofollow noreferrer">SSIM</a> and <a href="https://github.com/richzhang/PerceptualSimilarity" rel="nofollow noreferrer">LPIPS</a> losses for more information on these.</p>
  <p><a href="https://torchmetrics.readthedocs.io/en/latest/references/modules.html" rel="nofollow noreferrer">TorchMetrics</a> may be a good place to look implementations and available metrics.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-10-05T14:04:06.073Z
Answers.LastActivityDate: 2021-10-05T14:04:06.073Z
Answers.UserDisplayName: a crazy Minion
Body: <p>I have a segmentation which outputs only one channel image (2 class segmentation). I have used dice score for most of the time, but now higher powers in my team want me to expand evaluation metrics for segmentation model (if it's even possible). I have done some research and as far as right now I have found mainly that everybody uses dice score, and sometimes pixel to pixel binary accuracy, but for the latter seems not the best idea.</p>
  <p>If anybody knows something exciting or useful, I'd be glad to hear from them.</p>
  
Tags: <deep-learning><image-segmentation><metric>
Title: Aside from dice score, what other good metrics are used to evaluate segmentation models?
UserDisplayName: Artūras Drūteika
ThreadText: # Aside from dice score, what other good metrics are used to evaluate segmentation models?
  
  (Asked by: username_0 on 2021-08-03)
  
  I have a segmentation which outputs only one channel image (2 class segmentation). I have used dice score for most of the time, but now higher powers in my team want me to expand evaluation metrics for segmentation model (if it's even possible). I have done some research and as far as right now I have found mainly that everybody uses dice score, and sometimes pixel to pixel binary accuracy, but for the latter seems not the best idea.
  
  If anybody knows something exciting or useful, I'd be glad to hear from them.
  
  **Tags:** deep-learning, image-segmentation, metric
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2021-10-05. Score: 2 | Accepted answer
  
  Typical metrics used with segmentation problems are Recall, Precision and the F1 Score (similar or the same as the Dice score depending on the definition used). These can be evaluated per class or for all classes together, commonly referred to as micro and macro averages.
  
  Taking it further, you may wish to have a metric more robust to changes in the threshold. Here the Area under the Curve ([AUC](https://towardsdatascience.com/understanding-auc-roc-curve-68b2303cc9c5)) metric is commonly used.
  
  For a more sophisticated analysis you may also be interested in perceptual losses. These quantify how similar an image looks as perceived by people. This is particularly useful if say the shape of the prediction is important but small shifts or scaling does not matter. Have a look at [SSIM](https://en.wikipedia.org/wiki/Structural_similarity) and [LPIPS](https://github.com/richzhang/PerceptualSimilarity) losses for more information on these.
  
  [TorchMetrics](https://torchmetrics.readthedocs.io/en/latest/references/modules.html) may be a good place to look implementations and available metrics.
  
  
  ---
  
  ### Answer by username_1 on 2022-10-30. Score: 1
  
  I agree fully with @a crazy Minon's answer. I will just slightly expand on it and provide a couple of additional references.
  
  While Dice is a popular metric for evaluating segmentation, it is certainly not the only one. You are right in thinking that pixel accuracy is a poor choice of evaluation metric. The main issue is that it performs poorly when when there is class imbalance, which is often the case in imaging data.
  
  I will add that Intersection over Unions (IoU) is another metric that is frequently employed to evaluate segmentation performance. It is also known as the Jaccard Index. The articles ["Metrics to Evaluate your Semantic Segmentation Model"](https://towardsdatascience.com/metrics-to-evaluate-your-semantic-segmentation-model-6bcb99639aa2) and ["All the segmentation metrics!"](https://www.kaggle.com/code/yassinealouini/all-the-segmentation-metrics) provide good simplified introductions to various commonly used segmentation metrics. While Dice and IoU are similar and are positively correlated, they are not equivalent, as explained by this StackOverflow [answer](https://stats.stackexchange.com/questions/273537/f1-dice-score-vs-iou/276144#276144). New metrics are also being developed--such as the [Boundary Jaccard](http://www.irisa.fr/lagadic/pdf/2017_ppniv_fernandezmoral.pdf)--to overcome limitations of current metrics, and comparisons of these metrics have been published for specific applications (see example [ref](https://www.sciencedirect.com/science/article/pii/S0010482521002912), which lists 33 evaluation metrics for segmentation in Table 1).
  
  Finally, if your interest is really for one class, then accuracy, sensitivity, and specificity for segmenting that class alone can be useful metrics.
  
  The powers that be at your institution are wise in asking for multiple evaluation metrics because each metric has its limitations and no single metric can fully capture the performance of a segmentation network.
  
  
  ---
  
  ### Answer by username_2 on 2024-01-29. Score: 0
  
  <https://metrics-reloaded.dkfz.de/> -> Link to cite.
  
  This is an excellent site to see how to gauge which type of task and metric to use.
  
  Since you specifically want segmentation, just opt for that and have a look!
  
  Reference: <https://arxiv.org/abs/2206.01653>
  
  > **Comments:**
  > **username_3**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  
  ---
  
  