Machine Learning, NLP, Information Technology, Data, AI.
over an alphabet of size 2: The entropy is maximized for p=1/2(“uniform” distribution) and minimized for p=0or p=1(certainty). Notice also that for discrete random variables, the entropy cannot be negative: Entropy of a Bernoulli random variable X, as a function of X’s probability of being 1