Correcting for sample size in skewed data Empirically I have noticed that when data is skewed, and when for example interested in the 99th percentile of latencies, that it is difficult to compare experiments with different sample sizes
Imagine I run an experiment for 500 iterations and then afterwards for 2000 iterations, then the p99 when iter=2000 is simply higher
This is example descriptives data:
count    583.0000000000
mean     810.3430531732
std     1472.5106312310
min      112.0000000000
1%       166.6400000000
10%      234.0000000000
25%      274.5000000000
50%      373.0000000000
75%      779.5000000000
90%     1575.0000000000
99%     7404.4400000000
max    16479.0000000000

How can I make this data comparable between experiments correcting for the sample size? Is there some formula?

There are some nasty cases to think of: some probability distributions with a fat tail have an infinite mean value. One example is $P(X=x) = \frac{1}{x^2}$.
If you are interested in the tail, then big samples become all the more important.
