Essential Math, Editors Pick, Mathematics, Data Science, Machine Learning.
of the information of a distribution corresponds to the average of the information you’ll get. Following the formula of the expectation and the Shannon information, the entropy of the random variable X is defined as: The entropy gives you the average quantity of information that you need to encode