How large a sample do I need to be confident of a $\leq 2~\%$ incidence of feature in a population My specific context is detecting errors in automatic classification. Want to regularly manually review a subset of all events in one class, to ensure that the process putting them in that class is highly reliable. I’m wondering how few events I can sample and still get a solid estimate of reliability at the population level.
The method I’ve come across which so far seems most promising is Cochran’s formula. However, if I’m doing the math correctly it gives some results which seem very counterintuitive if not obviously wrong. For example, assuming 2% incidence of error and requiring 95% confidence in the conclusions, I calculate a sample size of 20 as follows.
\begin{align}
n &= \frac{Z^2(p)(q)}{e^2}\\&=\frac{1.6^2(0.98) (.02) }{0.05^2}\\&= 20.
\end{align}
Hard to believe that you could be highly confident about a 2% incidence in population with a sample size of 20. Using a confidence level of 98% gives a more reasonable sample size of ~200, but the tiny sample size suggested for 95% confidence makes me wonder about the method – say if there are some prerequisite assumptions for using it that I’m not aware of.
In case it's relevant, the population size in this case is tens of thousands.
Tentative plan is to use a Z Proportion Hypothesis Test on the sample after it has been manually labelled, to flag instances where we can be confident that error rate is above 2%. More fundamentally, I want to be confident that error rate is below 2% in population when less than 2% error is found in sample – that is the key requirement on sample size.
Should I use Cochran’s formula to find that sample size? Or is there a better method for this case?

One approach is visualizing the distribution of observed sample error rate for the two population error rates of interest: the expected population error rate, and error rate which we want to guarantee population is below. By doing that for several candidate sample sizes, we can see where what sample size is required to make the distributions overlap little enough that we can consistently statistically test to reject the hypothesis that "population error rate" >= "guarantee below error rate".
This is depicted below for the expected error rate 1.2% and "guarantee error rate" 2%.

If the true population error rate is 1.2%, and a random sample of 1250 cases is drawn, 80% of the time it will be possible to reject the hypothesis that population error rate is >= 2%.
In fact, in my practical use case, it's not good enough to have a 20% chance of failing to correctly reject the hypothesis that population error rate is higher that "guarantee" rate. A 5% chance of failing to correctly rejected the hypothesis is more reasonable, which requires a sample size of 2300.

Some details on method:

*

*The distributions are calculated with equation
\begin{align}p_x = {sample\_size \choose x} \cdot population\_proportion^x \cdot (1 - population\_proportion)^{sample\_size - x}
\end{align}
where $p_x$ is the probability of observing x error cases in a sample, given sample size and population error rate.


*The threshold bounding 95% of samples -- assuming expected population error rate -- is found by cumulatively summing the probability of each number of error cases observed in sample, and then taking the point at which that cumulative sum equals 0.95. Black line on the graphs.


*The statistical test used was one-sided one-sample z test for proportions. The key point is largest number of error cases, less than "guarantee error rate", where p value of test is less than 0.05. Red line on the graphs.
Overall, this methodology gives the same output as power analysis, or as "planning for precision". I just found those ways somewhat like black magic when they were pure math outputting a number. To me, visualizing the relevant distributions helped with intuitively understanding why these sample sizes are adequate, so perhaps this perspective will also be useful for others.
Thank you so much to whuber, kqr, and dipetkov for explaining the considerations involved and spelling out the power analysis and "planning for precision" methods.
