I think it comes down to what @KFrank explained.
The mean and stddev in a categorical distribution doesn’t make sense, as there is no innate ordering in the values.
The example of images makes it clear. Alternatively, we could also use an example for words.
If we assign each word from a text to a label, what would be the “mean word” and what would the standard deviation mean in this context?