Answers.Body: <p>It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-19T10:32:57.197Z
Answers.LastActivityDate: 2018-10-19T10:32:57.197Z
Answers.UserDisplayName: user19149
Body: <p>I'm creating a Decision Tree and at the very root level itself, I'm getting <strong>negative Information Gain</strong>. </p>
  
  <p>As per my knowledge, Information Gain is always <code>&gt; 0</code>....</p>
  
  <p>Any explanation....please....</p>
  
  <p>Please look at the node below.....</p>
  
  <pre><code>    [69+,42-]
       /   \        
      /     \                       
  [56+,33-]  [11+,11-]
  
  IG = H([69+, 42-]) - H([56+,33-],[11+,11-]) 
  
     = {(-69/111 * lg(69/111) -42/111 * lg(42/111)) 
        - 89/111 * ( -56/89*lg(56/89) -33/89*lg(33/89)) 
        - 22/111 * ( -11/22*lg(11/22) -11/22*lg(11/22))}
  
     = -0.004
  </code></pre>
  
  <p>It turns out that for every feature, the <code>IG &lt; 0</code>.</p>
  
  <p>What should I do to decide the feature at the root node?</p>
  
  <p>Thanks</p>
  
Tags: <machine-learning><decision-theory>
Title: Getting Negative Information Gain (using Entropy as a measure of disorder)
UserDisplayName: imflash217
ThreadText: # Getting Negative Information Gain (using Entropy as a measure of disorder)
  
  (Asked by: username_0 on 2017-12-18)
  
  I'm creating a Decision Tree and at the very root level itself, I'm getting **negative Information Gain**.
  
  As per my knowledge, Information Gain is always `> 0`....
  
  Any explanation....please....
  
  Please look at the node below.....
  
  ```
      [69+,42-]
       /   \        
      /     \                       
  [56+,33-]  [11+,11-]
  
  IG = H([69+, 42-]) - H([56+,33-],[11+,11-]) 
  
     = {(-69/111 * lg(69/111) -42/111 * lg(42/111)) 
        - 89/111 * ( -56/89*lg(56/89) -33/89*lg(33/89)) 
        - 22/111 * ( -11/22*lg(11/22) -11/22*lg(11/22))}
  
     = -0.004
  
  ```
  
  It turns out that for every feature, the `IG < 0`.
  
  What should I do to decide the feature at the root node?
  
  Thanks
  
  **Tags:** machine-learning, decision-theory
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-10-19. Score: 2
  
  It is not possible to have a negative IG, your outcome is negative because of a computational mistake: initially you have 69 positive instances, and after creating 2 children the sum drops to 56+11=67. Set is therefore less pure -> IG is negative.
  
  
  ---
  
  