Philosophy, Information Theory, Energy, Thermodynamics.
To measure the purity of a sample subset, the algorithm needs a quantitative measure to objectively choose which feature to split on at every given node. This is where entropy comes in. At each step, the algorithm calculates which attribute will show the highest information gain, then decides on