Are there better measures of entropy Related question here
I am trying to measure the uniformity of multimodal distributions and am looking into using entropy.
I would like a measure of entropy that is higher for the first distribution than the second.

However, Shannon entropy is order independent (invariant under permutations) so the above distributions have the same Shannon entropy.
Is there anyway to capture clumping in the measure of entropy?
My thinking is that under the heat equation, it should take longer for the second distribution's Shannon entropy to become asymptotic so it should have lower entropy.

Let there be 6 bins evenly filled out of 12. Shannon entropy doesn't distinguish between the case A where all even bins are filled vs case B where bins [0,3) and [9, 12) are filled




index
0
1
2
3
4
5
6
7
8
9
10
11




A
0
1/6
0
1/6
0
1/6
0
1/6
0
1/6
0
1/6


B
1/6
1/6
1/6
0
0
0
0
0
0
1/6
1/6
1/6





Right--but so what? The indexes of the bins are irrelevant. If they are relevant to you, then you do not need entropy; you need another concept. But in that case, what are you trying to characterize? @whuber

I though this might have been studied and you could tell me?

Shannon's entropy distinguishes between those two scenarios:
Two bins:
$$- \sum_{i=1}^2 \frac{1}{2} \log \frac{1}{2} = \log 2$$
Ten bins:
$$- \sum_{i=1}^{10} \frac{1}{10} \log \frac{1}{10} = \log 10$$
Finite $n$ bins:
$$- \sum_{i=1}^{n} \frac{1}{n} \log \frac{1}{n} = \log n$$
And note that $\log 10 > \log 2$. And since logarithms are monotonic functions, we know that more bins means a bigger number. I.e. $m > n \implies \log m > \log n$.

You are right that Shannon's entropy is invariant to the permutations of the bins. If the entropy depended on permutation, then it would distinguish between different versions of your two examples. You have ten bins in the first example. Such a permutation-varying entropy would consider the order of bins (1, 2, 3, 4, 5, 6, 7, 8, 9, 10) different from (1, 2, 3, 4, 5, 6, 7, 8, 10, 9), and yet different from (1, 2, 3, 4, 5, 6, 7, 9, 8, 10),.... for all $10! = 3628800$ permutations of ten bins.

Re: 10/20 bins filled example:
You might be interested in normalizing the entropy to the max possible entropy of 20 bins.
Let us assume there is 20 outcomes, then for any uniform choice on ten of those bins we will have a normalized entropy of:
$$0 \leq \frac{H_{10}}{H_{\max}} = \frac{\log 10}{ \log 20} = \log_{20}(10) \approx 0.768621786840241 \leq 1$$


Is there anyway to capture clumping in the measure of entropy?

I'm not sure this is really what you want, but you could compute a sort of weighted entropy:
$$H_{w(x)} = - \sum_x [w(x)p(x)] \log [w(x)p(x)]$$
where $0 \leq w(x) \leq 1$ is some kind of weight based on the distances between bins which satisfies $\sum_x w(x) = 1$.
