Answers.Body: <p>Suppose you have data:</p>
  
  <pre><code>color  height  quality
  =====  ======  =======
  green  tall    good
  green  short   bad
  blue   tall    bad
  blue   short   medium
  red    tall    medium
  red    short   medium
  </code></pre>
  
  <p>To calculate the entropy for quality in this example:</p>
  
  <pre><code>X  = {good, medium, bad}
  x1 = {good}, x2 = {bad}, x3 = {medium}
  </code></pre>
  
  <p>Probability of each x in X:</p>
  
  <pre><code>p1 = 1/6 = 0.16667
  p2 = 2/6 = 0.33333
  p3 = 3/6 = 0.5
  </code></pre>
  
  <p>for which logarithms are:</p>
  
  <pre><code>log2(p1) = -2.58496
  log2(p2) = -1.58496
  log2(p3) = -1.0
  </code></pre>
  
  <p>and therefore entropy for the set is:</p>
  
  <pre><code>H(X) = - (0.16667 * -2.58496) - (0.33333 * -1.58496) - (0.5 * -1.0)
       = 1.45915
  </code></pre>
  
  <p>by the formula in the question.</p>
  
  <p>Remaining tasks are to iterate this process for each attribute to form the nodes of the tree.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-03T20:47:16.727Z
Answers.Text: how are p1, p2, and p3 calculated?
Answers.UserDisplayName: MSB
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-10T18:01:06.027Z
Answers.Text: `p1 = 1/6 = 0.16667`
  `p2 = 2/6 = 0.33333`
  `p3 = 3/6 = 0.5`
Answers.UserDisplayName: dynrepsys
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2016-11-22T22:02:05.037Z
Answers.LastActivityDate: 2016-11-22T22:02:05.037Z
Answers.UserDisplayName: dynrepsys
Answers.Body: <p>Very nice example that makes perfect sense to me. To get the value into expected range [0..1] <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Efficiency_(normalized_entropy)" rel="nofollow noreferrer">normalization</a> is needed.</p>
  <p><span class="math-container">$H(S)_{norm}=\frac{H(S)}{log_2(|S|)}=\frac{H( \{good, bad, bad, medium, medium, medium\} )}{log_2(3)}=\frac{1.459147917027245}{1.584962500721156}=0.9206198357143052 \in [0..1]$</span></p>
  <p>✔️</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-28T15:07:39.290Z
Answers.LastActivityDate: 2022-12-29T15:51:34.320Z
Answers.LastEditDate: 2022-12-29T15:51:34.320Z
Answers.UserDisplayName: Christian
Body: <p>Here is the definition of the entropy</p>
  <p><span class="math-container">$$H(S)=-\sum_{x \in X} p(x) \log _{2} p(x)$$</span></p>
  <p><a href="https://en.wikipedia.org/wiki/ID3_algorithm#Entropy" rel="nofollow noreferrer">Wikipedia</a>'s description of entropy breaks down the formula, but I still don't know how to determine the values of <span class="math-container">$X$</span>, defined as</p>
  <blockquote>
  <p>The set of classes in <span class="math-container">$S$</span></p>
  </blockquote>
  <p>and <span class="math-container">$p(x)$</span>, defined as</p>
  <blockquote>
  <p>The proportion of the number of elements in class <span class="math-container">$x$</span> to the number of elements in set <span class="math-container">$S$</span>.</p>
  </blockquote>
  <p>Can anyone break this down further to explain how to find <span class="math-container">$p(x)$</span>?</p>
  
Tags: <machine-learning><classification><decision-trees><id3-algorithm><entropy>
Title: How to calculate the entropy in the ID3 decision tree algorithm?
UserDisplayName: Z. Reticulan
ThreadText: # How to calculate the entropy in the ID3 decision tree algorithm?
  
  (Asked by: username_0 on 2016-11-06)
  
  Here is the definition of the entropy
  
  $$H(S)=-\sum\_{x \in X} p(x) \log \_{2} p(x)$$
  
  [Wikipedia](https://en.wikipedia.org/wiki/ID3_algorithm#Entropy)'s description of entropy breaks down the formula, but I still don't know how to determine the values of $X$, defined as
  
  > The set of classes in $S$
  
  and $p(x)$, defined as
  
  > The proportion of the number of elements in class $x$ to the number of elements in set $S$.
  
  Can anyone break this down further to explain how to find $p(x)$?
  
  **Tags:** machine-learning, classification, decision-trees, id3-algorithm, entropy
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2016-11-22. Score: 3
  
  Suppose you have data:
  
  ```
  color  height  quality
  =====  ======  =======
  green  tall    good
  green  short   bad
  blue   tall    bad
  blue   short   medium
  red    tall    medium
  red    short   medium
  
  ```
  
  To calculate the entropy for quality in this example:
  
  ```
  X  = {good, medium, bad}
  x1 = {good}, x2 = {bad}, x3 = {medium}
  
  ```
  
  Probability of each x in X:
  
  ```
  p1 = 1/6 = 0.16667
  p2 = 2/6 = 0.33333
  p3 = 3/6 = 0.5
  
  ```
  
  for which logarithms are:
  
  ```
  log2(p1) = -2.58496
  log2(p2) = -1.58496
  log2(p3) = -1.0
  
  ```
  
  and therefore entropy for the set is:
  
  ```
  H(X) = - (0.16667 * -2.58496) - (0.33333 * -1.58496) - (0.5 * -1.0)
       = 1.45915
  
  ```
  
  by the formula in the question.
  
  Remaining tasks are to iterate this process for each attribute to form the nodes of the tree.
  
  > **Comments:**
  > **username_2**: how are p1, p2, and p3 calculated?  
  > **username_1**: `p1 = 1/6 = 0.16667` `p2 = 2/6 = 0.33333` `p3 = 3/6 = 0.5`  
  
  ---
  
  ### Answer by username_3 on 2022-12-28. Score: 0
  
  Very nice example that makes perfect sense to me. To get the value into expected range [0..1] [normalization](https://en.wikipedia.org/wiki/Entropy_(information_theory)#Efficiency_(normalized_entropy)) is needed.
  
  $H(S)\_{norm}=\frac{H(S)}{log\_2(|S|)}=\frac{H( \{good, bad, bad, medium, medium, medium\} )}{log\_2(3)}=\frac{1.459147917027245}{1.584962500721156}=0.9206198357143052 \in [0..1]$
  
  ✔️
  
  
  ---
  
  