<p>Suppose you have data:</p>

<pre><code>color  height  quality
=====  ======  =======
green  tall    good
green  short   bad
blue   tall    bad
blue   short   medium
red    tall    medium
red    short   medium
</code></pre>

<p>To calculate the entropy for quality in this example:</p>

<pre><code>X  = {good, medium, bad}
x1 = {good}, x2 = {bad}, x3 = {medium}
</code></pre>

<p>Probability of each x in X:</p>

<pre><code>p1 = 1/6 = 0.16667
p2 = 2/6 = 0.33333
p3 = 3/6 = 0.5
</code></pre>

<p>for which logarithms are:</p>

<pre><code>log2(p1) = -2.58496
log2(p2) = -1.58496
log2(p3) = -1.0
</code></pre>

<p>and therefore entropy for the set is:</p>

<pre><code>H(X) = - (0.16667 * -2.58496) - (0.33333 * -1.58496) - (0.5 * -1.0)
     = 1.45915
</code></pre>

<p>by the formula in the question.</p>

<p>Remaining tasks are to iterate this process for each attribute to form the nodes of the tree.</p>

CC BY-SA 4.0
2022-07-03T20:47:16.727Z
how are p1, p2, and p3 calculated?
MSB
CC BY-SA 4.0
2022-07-10T18:01:06.027Z
`p1 = 1/6 = 0.16667`
`p2 = 2/6 = 0.33333`
`p3 = 3/6 = 0.5`
dynrepsys
CC BY-SA 3.0
2016-11-22T22:02:05.037Z
2016-11-22T22:02:05.037Z
dynrepsys
<p>Very nice example that makes perfect sense to me. To get the value into expected range [0..1] <a href="https://en.wikipedia.org/wiki/Entropy_(information_theory)#Efficiency_(normalized_entropy)" rel="nofollow noreferrer">normalization</a> is needed.</p>
<p><span class="math-container">$H(S)_{norm}=\frac{H(S)}{log_2(|S|)}=\frac{H( \{good, bad, bad, medium, medium, medium\} )}{log_2(3)}=\frac{1.459147917027245}{1.584962500721156}=0.9206198357143052 \in [0..1]$</span></p>
<p>✔️</p>

CC BY-SA 4.0
2022-12-28T15:07:39.290Z
2022-12-29T15:51:34.320Z
2022-12-29T15:51:34.320Z
Christian
<p>Here is the definition of the entropy</p>
<p><span class="math-container">$$H(S)=-\sum_{x \in X} p(x) \log _{2} p(x)$$</span></p>
<p><a href="https://en.wikipedia.org/wiki/ID3_algorithm#Entropy" rel="nofollow noreferrer">Wikipedia</a>'s description of entropy breaks down the formula, but I still don't know how to determine the values of <span class="math-container">$X$</span>, defined as</p>
<blockquote>
<p>The set of classes in <span class="math-container">$S$</span></p>
</blockquote>
<p>and <span class="math-container">$p(x)$</span>, defined as</p>
<blockquote>
<p>The proportion of the number of elements in class <span class="math-container">$x$</span> to the number of elements in set <span class="math-container">$S$</span>.</p>
</blockquote>
<p>Can anyone break this down further to explain how to find <span class="math-container">$p(x)$</span>?</p>

<machine-learning><classification><decision-trees><id3-algorithm><entropy>
How to calculate the entropy in the ID3 decision tree algorithm?
Z. Reticulan
# How to calculate the entropy in the ID3 decision tree algorithm?

(Asked by: username_0 on 2016-11-06)

Here is the definition of the entropy

$$H(S)=-\sum\_{x \in X} p(x) \log \_{2} p(x)$$

[Wikipedia](https://en.wikipedia.org/wiki/ID3_algorithm#Entropy)'s description of entropy breaks down the formula, but I still don't know how to determine the values of $X$, defined as

> The set of classes in $S$

and $p(x)$, defined as

> The proportion of the number of elements in class $x$ to the number of elements in set $S$.

Can anyone break this down further to explain how to find $p(x)$?

**Tags:** machine-learning, classification, decision-trees, id3-algorithm, entropy


---

## 2 answers

### Answer by username_1 on 2016-11-22. Score: 3

Suppose you have data:
color  height  quality
=====  ======  =======
green  tall    good
green  short   bad
blue   tall    bad
blue   short   medium
red    tall    medium
red    short   medium

To calculate the entropy for quality in this example:
X  = {good, medium, bad}
x1 = {good}, x2 = {bad}, x3 = {medium}

Probability of each x in X:
p1 = 1/6 = 0.16667
p2 = 2/6 = 0.33333
p3 = 3/6 = 0.5

for which logarithms are:
log2(p1) = -2.58496
log2(p2) = -1.58496
log2(p3) = -1.0

and therefore entropy for the set is:
H(X) = - (0.16667 * -2.58496) - (0.33333 * -1.58496) - (0.5 * -1.0)
     = 1.45915

by the formula in the question.
Remaining tasks are to iterate this process for each attribute to form the nodes of the tree.

> **Comments:**
> **username_2**: how are p1, p2, and p3 calculated?  
> **username_1**: `p1 = 1/6 = 0.16667` `p2 = 2/6 = 0.33333` `p3 = 3/6 = 0.5`  

---

### Answer by username_3 on 2022-12-28. Score: 0

Very nice example that makes perfect sense to me. To get the value into expected range [0..1] [normalization](https://en.wikipedia.org/wiki/Entropy_(information_theory)#Efficiency_(normalized_entropy)) is needed.

$H(S)\_{norm}=\frac{H(S)}{log\_2(|S|)}=\frac{H( \{good, bad, bad, medium, medium, medium\} )}{log\_2(3)}=\frac{1.459147917027245}{1.584962500721156}=0.9206198357143052 \in [0..1]$

✔️


---

