In information theory and statistics, Kullback's inequality is a lower bound on the Kullback–Leibler divergence expressed in terms of the large deviations rate function. If "P" and "Q" are probability distributions on the real line, such that "P" is absolutely continuous with respect to "Q", i.e. "P" « "Q", and whose first moments exist, then
formula_1
where formula_2 is the rate function, i.e. the convex conjugate of the cumulant-generating function, of formula_3, and formula_4 is the first moment of formula_5
The Cramér–Rao bound is a corollary of this result.
Proof.
Let "P" and "Q" be probability distributions (measures) on the real line, whose first moments exist, and such that "P" « "Q". Consider the natural exponential family of "Q" given by
formula_6
for every measurable set "A", where formula_7 is the moment-generating function of "Q". (Note that "Q"0 = "Q".) Then
formula_8
By Gibbs' inequality we have formula_9 so that
formula_10
Simplifying the right side, we have, for every real "θ" where formula_11
formula_12
where formula_4 is the first moment, or mean, of "P", and formula_14 is called the cumulant-generating function. Taking the supremum completes the process of convex conjugation and yields the rate function:
formula_15
Corollary: the Cramér–Rao bound.
Start with Kullback's inequality.
Let "X"θ be a family of probability distributions on the real line indexed by the real parameter θ, and satisfying certain regularity conditions. Then
formula_16
where formula_17 is the convex conjugate of the cumulant-generating function of formula_18 and formula_19 is the first moment of formula_20
Left side.
The left side of this inequality can be simplified as follows:
formula_21
which is half the Fisher information of the parameter "θ".
Right side.
The right side of the inequality can be developed as follows:
formula_22
This supremum is attained at a value of "t"=τ where the first derivative of the cumulant-generating function is formula_23 but we have formula_24 so that
formula_25
Moreover,
formula_26
Putting both sides back together.
We have:
formula_27
which can be rearranged as:
formula_28