Answers.Body: <p>Please, take a look at <a href="https://arxiv.org/ftp/arxiv/papers/1405/1405.2061.pdf" rel="nofollow noreferrer">Understanding Shannon's Entropy metric for Information</a>. </p>
  
  <p>The answer for the minus sign is in section 6. The probability logs are less than or equal to <span class="math-container">$0$</span>, so the minus sign guarantees that information (entropy) is always greater than or equal to <span class="math-container">$0$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2017-09-27T13:32:13.553Z
Answers.LastActivityDate: 2019-11-19T15:56:58.347Z
Answers.LastEditDate: 2019-11-19T15:56:58.347Z
Answers.UserDisplayName: xboard
Body: <p>What is the intuition behind the following entropy formula used in the ID3 algorithm?</p>
  
  <p><span class="math-container">$$
  \text{info}(D) = -\sum_{i=1}^m p_i \log_2(p_i)
  $$</span></p>
  
Tags: <machine-learning><math><decision-trees><id3-algorithm>
Title: What is the intuition behind the entropy formula used in the ID3 algorithm?
UserDisplayName: DRV
ThreadText: # What is the intuition behind the entropy formula used in the ID3 algorithm?
  
  (Asked by: username_0 on 2017-09-27)
  
  What is the intuition behind the following entropy formula used in the ID3 algorithm?
  
  $$
  \text{info}(D) = -\sum\_{i=1}^m p\_i \log\_2(p\_i)
  $$
  
  **Tags:** machine-learning, math, decision-trees, id3-algorithm
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-09-27. Score: 2 | Accepted answer
  
  Please, take a look at [Understanding Shannon's Entropy metric for Information](https://arxiv.org/ftp/arxiv/papers/1405/1405.2061.pdf).
  
  The answer for the minus sign is in section 6. The probability logs are less than or equal to $0$, so the minus sign guarantees that information (entropy) is always greater than or equal to $0$.
  
  
  ---
  
  