Answers.Body: <p>In machine learning, the <a href="https://en.wikipedia.org/wiki/Precision_and_recall" rel="nofollow noreferrer">accuracy</a> is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives (<span class="math-container">$\mathrm {TP}$</span>) and true negatives (<span class="math-container">$\mathrm {TN}$</span>), so the usual formula to calculate the accuracy is the following one (your first one).</p>
  
  <p><span class="math-container">\begin{align}
  \text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0}
  \end{align}</span></p>
  
  <p>The following formula corresponds to the <a href="https://en.wikipedia.org/wiki/Precision_and_recall" rel="nofollow noreferrer">threat score (<span class="math-container">$\mathrm{TS}$</span>) or critical success index (<span class="math-container">$\mathrm {CSI}$</span>)</a>.</p>
  
  <p><span class="math-container">\begin{align}
  {\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1}
  \end{align}</span></p>
  
  <p>In section 7 of the paper <a href="https://www.researchgate.net/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning" rel="nofollow noreferrer">A Literature Survey on Algorithms for Multi-label Learning</a>, it is written</p>
  
  <blockquote>
    <p>In traditional classification such as multi-class problems, <strong>accuracy</strong> is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in <strong>multi-label classification</strong>, predictions for an instance is a set of labels and, therefore, the prediction can be <strong>fully correct</strong>, <strong>partially correct (with different levels of correctness)</strong> or <strong>fully incorrect</strong>. None of these existing evaluation metrics capture such notion in their original form.</p>
  </blockquote>
  
  <p>Then they say</p>
  
  <blockquote>
    <p>Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: <strong>evaluating partitions</strong>, <strong>evaluating ranking</strong> and <strong>using label hierarchy</strong>. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes
    are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.</p>
  </blockquote>
  
  <p>In section 7.1, the authors state</p>
  
  <blockquote>
    <p>To capture the notion of <strong>partially correct</strong>, one strategy is to <strong>evaluate the average difference between the predicted labels and the actual labels for each test example</strong>, and then average over all examples in the test set. This approach is called <strong>example-based evaluations</strong>.</p>
  </blockquote>
  
  <p>To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.423.3142&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Discriminative Methods for Multi-labeled Classification</a>)</p>
  
  <p><span class="math-container">\begin{align}
  \frac{1}{n} \sum_{i=1}^{n} \frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|} \label{2} \tag{2}
  \end{align}</span></p>
  
  <p>where <span class="math-container">$Y_i$</span> is the true set of labels and <span class="math-container">$Z_i$</span> the predicted set of labels for the single instance (or observation) <span class="math-container">$i$</span>, so <span class="math-container">$\frac{|Y_i \cap Z_i|}{|Y_i \cup Z_i|}$</span> is the accuracy for the instance <span class="math-container">$i$</span>. If <span class="math-container">$|Y_i|=1$</span> and <span class="math-container">$|Z_i|=1$</span>, then we have a single-label classification problem.</p>
  
  <p>The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T17:09:32.907Z
Answers.Text: In equation.1, though there is no notion of multiple labels, I can consider confusion matrix for vector of predicted  and vector of actual  to calculate TS(Theta Score) and if I average over all the examples or instances isn't it same as equation.2?
Answers.UserDisplayName: Stephen Philip
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:12:04.280Z
Answers.Text: @StephenPhilip I don't think so. $Y_i$ and $Z_i$ should be sets (rather than vectors).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:32:42.357Z
Answers.Text: Yes,  and   are sets, just to disambiguate I was using programming language terminology(C++ uses generally vectors are used to store as arrays, in python lists are used), but my doubt still persists, isn't eq.1 is representation of eq.2 with average over all the observations.
Answers.UserDisplayName: Stephen Philip
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:40:12.127Z
Answers.Text: @StephenPhilip I don't understand where you see the equivalence between equation 1 and 2, but maybe I am blind now. Does $|Y_i \cap Z_i|$ represent only the **number** of true positives (i.e. the numerator of equation 1)? $Y_i \cap Z_i$ is the intersection between two sets. It doesn't matter the order of the elements, but only whether the elements are or not in the sets. When you're counting the true positives, the order matters (i.e. you should compare the labels of instance $i$ with the corresponding predictions $i$).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:42:06.837Z
Answers.Text: Btw, you also have sets in Python and C++, you don't have to use lists or vectors, which represent ordered sequences.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T18:45:13.853Z
Answers.Text: Thanks for making a comment on Python and C++ I'll make a note of it. Coming back.
  
  if ∩ =  (intersection)  is right way to understand, in a set of [0,0,1,1] ∩ [0,1,0,1] the intersection(∩) here is acting as XNOR operation but according to most of the observed explanations I found that it is an AND operation and this is inconsistent with eq.2
Answers.UserDisplayName: Stephen Philip
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-02-21T13:13:10.323Z
Answers.LastActivityDate: 2020-02-21T18:08:30.167Z
Answers.LastEditDate: 2020-02-21T18:08:30.167Z
Answers.UserDisplayName: nbro
Body: <p>Some sources consider the true negatives (TN) when computing the accuracy, while some don't.</p>
  
  <p>Source 1:
  <a href="https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b" rel="nofollow noreferrer">https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b</a></p>
  
  <p><a href="https://i.sstatic.net/qZ1dN.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/qZ1dN.png" alt="Considers TN"></a></p>
  
  <p>Source 2:<a href="https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf" rel="nofollow noreferrer">https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf</a></p>
  
  <p><a href="https://i.sstatic.net/v8XPx.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/v8XPx.png" alt="enter image description here"></a> </p>
  
  <p>which can be translated as </p>
  
  <p><a href="https://i.sstatic.net/ZNOum.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZNOum.png" alt="enter image description here"></a></p>
  
  <p>which one of these must be considered for my multi-label model.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-21T10:16:42.287Z
Comments.Text: There are many different metrics for all sorts of features; generally there is no right or wrong, and it depends on your application.  If you have multiple labels, you probably don't have TN, so the second one looks to be a better fit.
Comments.UserDisplayName: Oliver Mason
Tags: <machine-learning><classification><accuracy><metric><multi-label-classification>
Title: Why is there more than one way of calculating the accuracy?
UserDisplayName: Stephen Philip
ThreadText: # Why is there more than one way of calculating the accuracy?
  
  (Asked by: username_0 on 2020-02-21)
  
  Some sources consider the true negatives (TN) when computing the accuracy, while some don't.
  
  Source 1:
  <https://medium.com/greyatom/performance-metrics-for-classification-problems-in-machine-learning-part-i-b085d432082b>
  
  [![Considers TN](https://i.sstatic.net/qZ1dN.png)](https://i.sstatic.net/qZ1dN.png)
  
  Source 2:<https://www.researchgate.net/profile/Mohammad_Sorower/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning/links/58d1864392851cf4f8f4b72a/A-Literature-Survey-on-Algorithms-for-Multi-label-Learning.pdf>
  
  [![enter image description here](https://i.sstatic.net/v8XPx.png)](https://i.sstatic.net/v8XPx.png)
  
  which can be translated as
  
  [![enter image description here](https://i.sstatic.net/ZNOum.png)](https://i.sstatic.net/ZNOum.png)
  
  which one of these must be considered for my multi-label model.
  
  **Tags:** machine-learning, classification, accuracy, metric, multi-label-classification
  > **Comments:**
  > **username_1**: There are many different metrics for all sorts of features; generally there is no right or wrong, and it depends on your application.  If you have multiple labels, you probably don't have TN, so the second one looks to be a better fit.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-02-21. Score: 4 | Accepted answer
  
  In machine learning, the [accuracy](https://en.wikipedia.org/wiki/Precision_and_recall) is usually defined as the number of correct predictions divided by the total number of predictions. The correct predictions are the true positives ($\mathrm {TP}$) and true negatives ($\mathrm {TN}$), so the usual formula to calculate the accuracy is the following one (your first one).
  
  \begin{align}
  \text{Accuracy}=\frac {\mathrm {TP} + \mathrm {TN}}{\mathrm {TP} + \mathrm {TN} + \mathrm {FP} + \mathrm {FN}} \label{0}\tag{0}
  \end{align}
  
  The following formula corresponds to the [threat score ($\mathrm{TS}$) or critical success index ($\mathrm {CSI}$)](https://en.wikipedia.org/wiki/Precision_and_recall).
  
  \begin{align}
  {\displaystyle \mathrm {TS} ={\frac {\mathrm {TP} }{\mathrm {TP} +\mathrm {FN} +\mathrm {FP} }}} \label{1} \tag{1}
  \end{align}
  
  In section 7 of the paper [A Literature Survey on Algorithms for Multi-label Learning](https://www.researchgate.net/publication/266888594_A_Literature_Survey_on_Algorithms_for_Multi-label_Learning), it is written
  
  > In traditional classification such as multi-class problems, **accuracy** is the most common evaluation criteria. Additionally, there exists a set of standard evaluation metrics that includes precision, recall, F-measure, and ROC area defined for single-label multi-class classification problems. However, in **multi-label classification**, predictions for an instance is a set of labels and, therefore, the prediction can be **fully correct**, **partially correct (with different levels of correctness)** or **fully incorrect**. None of these existing evaluation metrics capture such notion in their original form.
  
  Then they say
  
  > Depending on the target problem, evaluation measures for multi-label data can be grouped into at least three groups: **evaluating partitions**, **evaluating ranking** and **using label hierarchy**. The first one evaluates the quality of the classification into classes, the second one evaluates if the classes
  > are ranked in order relevance and the third one evaluates how effectively the learning system is able to take into account an existing hierarchical structure of the labels.
  
  In section 7.1, the authors state
  
  > To capture the notion of **partially correct**, one strategy is to **evaluate the average difference between the predicted labels and the actual labels for each test example**, and then average over all examples in the test set. This approach is called **example-based evaluations**.
  
  To capture the notion of partially incorrect in multi-label classification problems, they then use the following definition of accuracy (proposed in section 5.2 of the paper [Discriminative Methods for Multi-labeled Classification](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.423.3142&rep=rep1&type=pdf))
  
  \begin{align}
  \frac{1}{n} \sum\_{i=1}^{n} \frac{|Y\_i \cap Z\_i|}{|Y\_i \cup Z\_i|} \label{2} \tag{2}
  \end{align}
  
  where $Y\_i$ is the true set of labels and $Z\_i$ the predicted set of labels for the single instance (or observation) $i$, so $\frac{|Y\_i \cap Z\_i|}{|Y\_i \cup Z\_i|}$ is the accuracy for the instance $i$. If $|Y\_i|=1$ and $|Z\_i|=1$, then we have a single-label classification problem.
  
  The metric \ref{2} does not correspond to the metric \ref{1}. In \ref{2}, the accuracy is calculated for each example (or instance) and then we average all these accuracies, because, for each example, you may have that one predicted label is correct, two predicted labels are correct, and so on (also depending on the number of labels you need to predict for each of the examples). In \ref{1}, there is no notion of multiple labels.
  
  > **Comments:**
  > **username_0**: In equation.1, though there is no notion of multiple labels, I can consider confusion matrix for vector of predicted  and vector of actual  to calculate TS(Theta Score) and if I average over all the examples or instances isn't it same as equation.2?  
  > **username_2**: @StephenPhilip I don't think so. $Y_i$ and $Z_i$ should be sets (rather than vectors).  
  > **username_0**: Yes,  and   are sets, just to disambiguate I was using programming language terminology(C++ uses generally vectors are used to store as arrays, in python lists are used), but my doubt still persists, isn't eq.1 is representation of eq.2 with average over all the observations.  
  > **username_2**: @StephenPhilip I don't understand where you see the equivalence between equation 1 and 2, but maybe I am blind now. Does $|Y_i \cap Z_i|$ represent only the **number** of true positives (i.e. the numerator of equation 1)? $Y_i \cap Z_i$ is the intersection between two sets. It doesn't matter the order of the elements, but only whether the elements are or not in the sets. When you're counting the true positives, the order matters (i.e. you should compare the labels of instance $i$ with the corresponding predictions $i$).  
  > **username_2**: Btw, you also have sets in Python and C++, you don't have to use lists or vectors, which represent ordered sequences.  
  > **username_0**: Thanks for making a comment on Python and C++ I'll make a note of it. Coming back.  if ∩ =  (intersection)  is right way to understand, in a set of [0,0,1,1] ∩ [0,1,0,1] the intersection(∩) here is acting as XNOR operation but according to most of the observed explanations I found that it is an AND operation and this is inconsistent with eq.2  
  
  ---
  
  