 For instance, we can mistake an unfair coin for a fair one.According to this erroneous assumption, our uncertainty about the "j"-th event, or equivalently, the amount of information provided after observing the "j"-th event, is

The (assumed) average uncertainty of all possible events is then

On the other hand, the Shannon entropy of the probability distribution "p", defined by

is the real amount of uncertainty before observation.Therefore the difference between these two quantities

is a measure of the distinguishability of the two probability distributions "p" and "q".