Answers.Body: <p>Even with a binary classifier, one number does not fully represent the behaviour - the confusion matrix has three degrees of freedom. Even more with a multi-class problem, it is best to print our the whole confusion matrix. Then you can pick up problems like &quot;large class A is well classified, many Bs are wrongly classified as C, and the few Ds are wrongly assigned to A,B, or C&quot;.</p>
  <p>Even better, printing the confusion matrix helps you think about what the real business goals are: which of these errors matters most in practice?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-29T16:08:31.317Z
Answers.LastActivityDate: 2021-03-29T16:08:31.317Z
Answers.UserDisplayName: chrishmorris
Answers.Body: <p>Your intuition is correct. We do use other metrics for multi-label classification. The meaning of evaluation itself changes. Apart from grading the classifier on whether it classifies correctly or not, we also have to penalize it, if it chooses the wrong class appropriately. You could use the following metrics:</p>
  <ul>
  <li>micro/macro averaging Recall/Precision, etc.</li>
  <li>Hamming Loss</li>
  <li>Subset Accuracy</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-29T11:39:28.523Z
Answers.Text: Maybe you should briefly describe how they are calculated (or what's the intuition behind them) or, at least, provide a link to a research paper (or reliable article) that explains these metrics/measures.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-29T03:50:01.190Z
Answers.LastActivityDate: 2020-10-29T03:50:01.190Z
Answers.UserDisplayName: Saurav Maheshkar
Body: <p>Multi-label assignment is the task in machine learning to assign to each input value a set of categories from a fixed vocabulary where the categories need not be statistically independent, so precluding building a set of independent classifiers each classifying the inputs as belong to each of the categories or not.</p>
  <p>Machine learning also needs a measure by which the model may be evaluated. So this is the question how do we evaluate a multi-label classifier?</p>
  <p>We can’t use the normal recall,  accuracy and  F measures since they require a binary is it correct or not measure of each categorisation. Without such a measure we have no obvious means to evaluate models nor to measure concept drift.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-10-28T20:04:01.567Z
Comments.Text: I like this discussion of accuracy: https://stats.stackexchange.com/q/312780/247274. Two example of the strictly proper scoring rules that Kolassa discusses are crossentropy loss (log loss) and Brier score.
Comments.UserDisplayName: Dave
Tags: <reference-request><performance><metric><multi-label-classification>
Title: How do you measure multi-label classification accuracy?
UserDisplayName: Nick
ThreadText: # How do you measure multi-label classification accuracy?
  
  (Asked by: username_0 on 2020-10-28)
  
  Multi-label assignment is the task in machine learning to assign to each input value a set of categories from a fixed vocabulary where the categories need not be statistically independent, so precluding building a set of independent classifiers each classifying the inputs as belong to each of the categories or not.
  
  Machine learning also needs a measure by which the model may be evaluated. So this is the question how do we evaluate a multi-label classifier?
  
  We can’t use the normal recall, accuracy and F measures since they require a binary is it correct or not measure of each categorisation. Without such a measure we have no obvious means to evaluate models nor to measure concept drift.
  
  **Tags:** reference-request, performance, metric, multi-label-classification
  > **Comments:**
  > **username_1**: I like this discussion of accuracy: https://stats.stackexchange.com/q/312780/247274. Two example of the strictly proper scoring rules that Kolassa discusses are crossentropy loss (log loss) and Brier score.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2021-03-29. Score: 0
  
  Even with a binary classifier, one number does not fully represent the behaviour - the confusion matrix has three degrees of freedom. Even more with a multi-class problem, it is best to print our the whole confusion matrix. Then you can pick up problems like "large class A is well classified, many Bs are wrongly classified as C, and the few Ds are wrongly assigned to A,B, or C".
  
  Even better, printing the confusion matrix helps you think about what the real business goals are: which of these errors matters most in practice?
  
  
  ---
  
  ### Answer by username_3 on 2020-10-29. Score: 0
  
  Your intuition is correct. We do use other metrics for multi-label classification. The meaning of evaluation itself changes. Apart from grading the classifier on whether it classifies correctly or not, we also have to penalize it, if it chooses the wrong class appropriately. You could use the following metrics:
  
  * micro/macro averaging Recall/Precision, etc.
  * Hamming Loss
  * Subset Accuracy
  
  > **Comments:**
  > **username_4**: Maybe you should briefly describe how they are calculated (or what's the intuition behind them) or, at least, provide a link to a research paper (or reliable article) that explains these metrics/measures.  
  
  ---
  
  