Answers.Body: <p>Categorical just means that we will conduct <em>multiclass classification</em>. The output of the classifier is a binary vector. Each entry <span class="math-container">$x_i$</span> in the binary vector is a prediction whether or not the input is part of class <span class="math-container">$C_i$</span>.</p>
  <p>In that sense, categorical accuracy introduces nothing new: it is just the accuracy of a multiclass classifier. On the other hand, categorical cross-entropy refers to the <em><strong>joint entropy</strong></em>: <span class="math-container">$H(X_1, X_2) = - \sum{p(x_1,x_2)\log_2p(x_1, x_2)}$</span>, where each random variable <span class="math-container">$X_i$</span> expresses whether or not the input is to be classified into class <span class="math-container">$C_i$</span>. <a href="https://scholar.google.com/citations?user=MxxZkEcAAAAJ&amp;hl=en&amp;oi=sra" rel="nofollow noreferrer">Kevin Murphy's</a> book &quot;<a href="https://probml.github.io/pml-book/book1.html" rel="nofollow noreferrer">Probabilistic Machine Learning: An introduction</a>&quot; is a great source of reference for many topics of machine learning, including cross-entropy and joint entropy.</p>
  <p>The random variables could be mutually exclusive or not, depending on the problem.</p>
  <ul>
  <li>If they are mutually exclusive, we would allow only one of the classes to have value 1 during training and use the softmax activation function.</li>
  <li>If they are not mutually exclusive, we would allow multiple classes to have value 1 during training and use the sigmoid activation function.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T16:01:39.303Z
Answers.Text: what about the categorical accuracy metric?
Answers.UserDisplayName: user366312
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T21:00:01.817Z
Answers.Text: I edited the answer to mention categorical accuracy too. Basically, categorical accuracy is not a separate concept, so you would have a hard time finding it in literature.
Answers.UserDisplayName: devidduma
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-16T15:41:02.853Z
Answers.LastActivityDate: 2022-02-19T13:55:56.630Z
Answers.LastEditDate: 2022-02-19T13:55:56.630Z
Answers.UserDisplayName: devidduma
Body: <p>My Python source code uses TensorFlow and Keras to implement a neural network.</p>
  <p>The Keras source code uses something called &quot;categorical cross-entropy&quot; and &quot;categorical accuracy metric&quot;. I have searched a lot of books on NN theory, and no one talks about these two specific terms. Yes, they talk about &quot;cross-entropy&quot; and &quot;accuracy metric&quot; but there are no mentions of &quot;categorical ...&quot;.</p>
  <p><strong>N.B.</strong> These terms can be found only in the so-called &quot;Hands-on&quot; books.</p>
  <p>Can anyone please supply me with authentic references on these two specific terminologies?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-14T03:41:40.003Z
Comments.Text: Cross-posted: https://ai.stackexchange.com/q/34507/1794, https://cstheory.stackexchange.com/q/51092/5038.  Please [do not post the same question on multiple sites](https://meta.stackexchange.com/q/64068).
Comments.UserDisplayName: D.W.
Tags: <tensorflow><python><keras><accuracy><categorical-crossentropy>
Title: Where can I find authentic references on "categorical cross entropy" and "categorical accuracy metric"?
UserDisplayName: user366312
ThreadText: # Where can I find authentic references on "categorical cross entropy" and "categorical accuracy metric"?
  
  (Asked by: username_0 on 2022-02-13)
  
  My Python source code uses TensorFlow and Keras to implement a neural network.
  
  The Keras source code uses something called "categorical cross-entropy" and "categorical accuracy metric". I have searched a lot of books on NN theory, and no one talks about these two specific terms. Yes, they talk about "cross-entropy" and "accuracy metric" but there are no mentions of "categorical ...".
  
  **N.B.** These terms can be found only in the so-called "Hands-on" books.
  
  Can anyone please supply me with authentic references on these two specific terminologies?
  
  **Tags:** tensorflow, python, keras, accuracy, categorical-crossentropy
  
  > **Comments:**
  > **username_1**: Cross-posted: https://ai.stackexchange.com/q/34507/1794, https://cstheory.stackexchange.com/q/51092/5038.  Please [do not post the same question on multiple sites](https://meta.stackexchange.com/q/64068).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-16. Score: 2 | Accepted answer
  
  Categorical just means that we will conduct *multiclass classification*. The output of the classifier is a binary vector. Each entry $x\_i$ in the binary vector is a prediction whether or not the input is part of class $C\_i$.
  
  In that sense, categorical accuracy introduces nothing new: it is just the accuracy of a multiclass classifier. On the other hand, categorical cross-entropy refers to the ***joint entropy***: $H(X\_1, X\_2) = - \sum{p(x\_1,x\_2)\log\_2p(x\_1, x\_2)}$, where each random variable $X\_i$ expresses whether or not the input is to be classified into class $C\_i$. [Kevin Murphy's](https://scholar.google.com/citations?user=MxxZkEcAAAAJ&hl=en&oi=sra) book "[Probabilistic Machine Learning: An introduction](https://probml.github.io/pml-book/book1.html)" is a great source of reference for many topics of machine learning, including cross-entropy and joint entropy.
  
  The random variables could be mutually exclusive or not, depending on the problem.
  
  * If they are mutually exclusive, we would allow only one of the classes to have value 1 during training and use the softmax activation function.
  * If they are not mutually exclusive, we would allow multiple classes to have value 1 during training and use the sigmoid activation function.
  
  > **Comments:**
  > **username_0**: what about the categorical accuracy metric?  
  > **username_2**: I edited the answer to mention categorical accuracy too. Basically, categorical accuracy is not a separate concept, so you would have a hard time finding it in literature.  
  
  ---
  
  