Essential Math, Editors Pick, Mathematics, Data Science, Machine Learning.
information I(x) is measured in bits. Entropy Consider for instance a biased coin, where you have a probability of 0.8 of getting ‘heads’. Here is your distribution: you have a probability of 0.8 of getting ‘heads’ and a probability of 1–0.8 = 0.2 of getting ‘tails’. These probabilities are