Why is the entropy of a binary random variables maximised when its probability is 0.5? I've found the explanation that:
The entropy of a binary random (Bernoulli) variable is a function of its probability and maximum when its probability is 0.5 (when it has an entropy of 1 bit). Intuitively, if a measurement is always false (or always true) then we are not uncertain of its value. If instead it is true as often as not, then the uncertainty, and hence the entropy, is maximised.
While I can see how this argument makes sense in the extreme case, I don't intuitively see why this is true. If a binary random variable is 1 with 80% probability and 0 with 20% probability, then when it is 1 will we will not be very surprised. However, when it is 0 we will be quite surprised, and it seems plausible that these two effects could offset each other in some way.
Can anyone provide more intuition for this?


*

*The amount of potential information contained in a signal is what we call the entropy, usually denoted by $H$ and defined as follows:
$$H(X) = − \sum_X{P(X) \log P(X)}$$

*The "surprise" is (or you could say proportional to, but it doesn't matter) the inverse of the probability and you can see that the example you mentioned.

*So the entropy can be written as
$$ H(X) = \sum_X{P(X) \log\frac{1}{P(X)}} $$
which is a weighted average of the "surprise" i.e. if the probability is low, then there is a lot of surprise, and consequently  a lot of information is conveyed by telling you that X happened.

*Now to answer the question, YES the surprise is not at maximum when the probability is 0.5 and the event occurs or not (which makes sense since we would be equally surprised by the occurrence and non-occurrence) BUT the entropy/information is a weighted average over the two events

*Thus, entropy essentially measures the “average surprise” or “average uncertainty” of a random variable. If the distribution $P(X)$ is highly peaked around
one value, then we will rarely be surprised by this variable; hence it would be
incapable of conveying much information. If on the other hand $P(X)$ is uniformly distributed, then we will be most surprised on average by this variable;
hence it could potentially convey a lot of information.

