How to measure if a categorical distribution is concentrated in a very few bins I'm trying to think of a way to measure that a categorical distribution of any size is concentrated in only a few bins, so not uniform. The best way I can think of is checking entropy, but that's kind of hard to assess, because either it's close to a uniform distribution or it's not unless there's something I'm missing. I also heard about kurtosis, but that's more a metric of tailedness. I'd like that, for example, if I have 5 classes and 2 of the classes make up 80% of the distribution, then I have a metric that can reflect this.

I think that a metric like deviation from uniformity is best.  Here I have generated a series of categorical distributions (the plots below show bar charts of the probabilities).  Categories that are unobserved should be ignored.  For instance, the top two plots are both uniform, but the first with 20 categories and the second with 3.  It's possible one would want uniformity across 20 categories to be "more uniform" than 3. Note that here I assumed that all we have is the observed categories, and that any unobserved are ignored, which may not be realistic (e.g. in observing US states, you may want to measure the observed distribution to uniformity over all 50 states rather than just over the observed ones).
I calculated 3 different metrics:

*

*Kurtosis: taking a sample of 1000 values from 1:number of classes, according to the probability distribution (e.g., if 3 categories, then from 1,2,3); the probability distribution is sorted from high to low probability, so it is strictly non-increasing.  Note, Kurtosis is not sensitive to scale and location changes, which is good.  However, it is sensitive to the number of categories (i.e. unique values), for instance, in the first two plots.  Notice also that the first plot and the 6th have very similar kurtosis scores, even though the 6th has two high peaks.  However, it is possible kurtosis is not appropriate since the data is categorical, even though it is not sensitive to location and scale.

*Jensen Shannon: Here I calculate JS distance (https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.distance.jensenshannon.html) with base=2 (making it bounded in [0,1]), where the comparison distribution is uniform with number of nonzero categories observed.  This seems to be the best.  Uniform distributions have 0 distance.

*Entropy: Entropy, using base = number of nonzero observed categories; this makes it not sensitive to the number (e.g. any uniform has entropy=1).  The 5th and 8th have entropies close to 1.

It seems like JS distance is the most responsive to deviation from uniformity (in addition to being valid when one of the categories has 0 probability but you want to account for it).

