How percentile approximation works and why it's more useful than averages
Awhile ago I wrote a Python library called LiveStats[1] that computed any percentile for any amount of data using a fixed amount of memory per percentile. It uses an algorithm I found in an old paper[2] called P^2. It uses a polynomial  to find good approximations.<p>The reason I made this was an old Amazon interview question. The question was basically, "Find the median of a huge data set without sorting it," and the "correct" answer was to have a fixed size sorted buffer and randomly evict items from it and then use the median of the buffer. However, a candidate I was interviewing had a really brilliant insight: if we estimate the median and move it a small amount for each new data point, it would be pretty close. I ended up doing some research on this and found P^2, which is a more sophisticated version of that insight.<p>[1]: <a href="https://github.com/cxxr/LiveStats" rel="nofollow">https://github.com/cxxr/LiveStats</a><p>[2]: <a href="https://www.cs.wustl.edu/~jain/papers/ftp/psqr.pdf" rel="nofollow">https://www.cs.wustl.edu/~jain/papers/ftp/psqr.pdf</a>