Notion in information theory
In information theory, the limiting density of discrete points is an adjustment to the formula of Claude Shannon for differential entropy.
It was formulated by Edwin Thompson Jaynes to address defects in the initial definition of differential entropy.
Definition.
Shannon originally wrote down the following formula for the entropy of a continuous distribution, known as differential entropy:
 formula_1
Unlike Shannon's formula for the discrete entropy, however, this is not the result of any derivation (Shannon simply replaced the summation symbol in the discrete version with an integral), and it lacks many of the properties that make the discrete entropy a useful measure of uncertainty. In particular, it is not invariant under a change of variables and can become negative. In addition, it is not even dimensionally correct. Since formula_2 would be dimensionless, formula_3 must have units of formula_4, which means that the argument to the logarithm is not dimensionless as required.
Jaynes argued that the formula for the continuous entropy should be derived by taking the limit of increasingly dense discrete distributions. Suppose that we have a set of formula_5 discrete points formula_6, such that in the limit formula_7 their density approaches a function formula_8 called the "invariant measure":
formula_9
Typically, when this is written, the term formula_10 is omitted, as that would typically not be finite. So the actual common definition is
 formula_11
Where it is unclear whether or not the formula_10 term should be omitted, one could write
 formula_13
Notice that in Jaynes' formula, formula_14 is a probability density. For any finite formula_15 that formula_14 is a uniform density over the quantization of the continuous space that is used in the Riemann sum. In the limit, formula_14 is the continuous limiting density of points in the quantization used to represent the continuous variable formula_18.
Suppose one had a number format that took on formula_15 possible values, distributed as per formula_14. Then formula_21 (if formula_5 is large enough that the continuous approximation is valid) is the discrete entropy of the variable formula_18 in this encoding. This is equal to the average number of bits required to transmit this information, and is no more than formula_24. Therefore, formula_25 may be thought of as the amount of information gained by knowing that the variable formula_18 follows the distribution formula_3, and is not uniformly distributed over the possible quantized values, as would be the case if it followed formula_14. formula_29 is actually the (negative) Kullback–Leibler divergence from formula_8 to formula_3, which is thought of as the information gained by learning that a variable previously thought to be distributed as formula_14 is actually distributed as formula_3.
Jaynes' continuous entropy formula has the property of being invariant under a change of variables, provided that formula_8 and formula_35 are transformed in the same way. (This motivates the name "invariant measure" for "m".) This solves many of the difficulties that come from applying Shannon's continuous entropy formula. Jaynes himself dropped the formula_10 term as it was not relevant to his work (maximum entropy distributions), and it is somewhat awkward to have an infinite term in the calculation. Unfortunately, this cannot be helped if the quantization is made arbitrarily fine, as would be the case in the continuous limit. Note that formula_25 as defined here (without the formula_24 term) would always be non-positive, because a KL divergence would always be non-negative.
If it is the case that formula_14 is constant over some interval of size formula_40, and formula_3 is essentially zero outside that interval, then the limiting density of discrete points (LDDP) is closely related to the differential entropy formula_42:
 formula_43
References.
 ;templatestyles src="Reflist/styles.css" / ;