Are there better measures of entropy Related question here
I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.
I would like a measure of entropy that is higher for the first distribution than the second.

However, Shannon entropy is order independent (invariant under permutations) so the above distributions have the same Shannon entropy.
Is there anyway to capture clumping in the measure of entropy?
My thinking is that under the heat equation, it should take longer for the second distribution's Shannon entropy to become asymptotic so it should have lower entropy.

Let there be 6 bins evenly filled out of 12. Shannon entropy doesn't distinguish between the case A where all even bins are filled vs case B where bins [0,3) and [9, 12) are filled




index
0
1
2
3
4
5
6
7
8
9
10
11




A
0
1/6
0
1/6
0
1/6
0
1/6
0
1/6
0
1/6


B
1/6
1/6
1/6
0
0
0
0
0
0
1/6
1/6
1/6





Right--but so what? The indexes of the bins are irrelevant. If they are relevant to you, then you do not need entropy; you need another concept. But in that case, what are you trying to characterize? @whuber

I though this might have been studied and you could tell me?

Variance can be generalized to be about any point rather than just the mean (i.e. how spread out is the distribution from this point)
$$\mu \mapsto\sum p_i (x_i - \mu)^2.$$
We can also "invert" the variance to get a similarity measure for each point (how bunched is the distribution about this point)
$$\mu \mapsto \sum p_i e^{-(x_i - \mu)^2}.$$
Can then take the log of this to get a surprise function
$$\sigma :=\mu \mapsto -\log\sum p_i e^{-(x_i - \mu)^2}.$$
Entropy is then given by the formula
$$\prod \sigma(x_i)^{-p_i}$$
as explained by blog post 1 and 2 given by @kjetil answer. These links do a very good job of developing the language you need to think about entropy.


*

*Abundance:  the proportions in which the species occur (e.g. 50% grass, 30% clover, 20% daisies)

*Similarity:  the extent to which the species are related (e.g. an ecosystem made up of 50% snails and 50% slugs should probably be regarded as less diverse than one made up of 50% snails and 50% bats).


Shannon entropy is only a good measure when working with discrete metric spaces i.e. the distance between different "bins" is infininte. However, our bins have a natural distance between them (i.e. the difference of the indices) and so we need something more
