How large a sample do I need to be confident of a $\leq 2~\%$ incidence of feature in a population My specific context is detecting errors in automatic classification. Want to regularly manually review a subset of all events in one class, to ensure that the process putting them in that class is highly reliable. I’m wondering how few events I can sample and still get a solid estimate of reliability at the population level.
The method I’ve come across which so far seems most promising is Cochran’s formula. However, if I’m doing the math correctly it gives some results which seem very counterintuitive if not obviously wrong. For example, assuming 2% incidence of error and requiring 95% confidence in the conclusions, I calculate a sample size of 20 as follows.
\begin{align}
n &= \frac{Z^2(p)(q)}{e^2}\\&=\frac{1.6^2(0.98) (.02) }{0.05^2}\\&= 20.
\end{align}
Hard to believe that you could be highly confident about a 2% incidence in population with a sample size of 20. Using a confidence level of 98% gives a more reasonable sample size of ~200, but the tiny sample size suggested for 95% confidence makes me wonder about the method – say if there are some prerequisite assumptions for using it that I’m not aware of.
In case it's relevant, the population size in this case is tens of thousands.
Tentative plan is to use a Z Proportion Hypothesis Test on the sample after it has been manually labelled, to flag instances where we can be confident that error rate is above 2%. More fundamentally, I want to be confident that error rate is below 2% in population when less than 2% error is found in sample – that is the key requirement on sample size.
Should I use Cochran’s formula to find that sample size? Or is there a better method for this case?

Let's say what you're asking for is the standard error of the estimation of the proportion defective, and that these can be represented as independent draws from a binomial distribution.
This standard error is $\sqrt{\frac{p(1-p)}{n}}$ where $p$ is the measured proportion defective. For 95 % one-sided confidence, we want to see a measurement at least 1.645 standard errors away from 2 %. Thus, the full equation we want to solve is
$$p + 1.645\sqrt{\frac{p(1-p)}{n}} = 0.02.$$
As you can see, the sample size required actually depends on what proportion defective you expect to find. Here are some examples:

*

*p = 0.5 %, n = 60

*p = 1.0 %, n = 268

*p = 1.5 %, n = 1600

*p = 1.8 %, n = 11958

As you can probably intuit, the closer the observed proportion is to 2 %, the more data you need to be really sure it's not greater than 2 %.

Note that this assumes the total population is practically infinite. You haven't specified the size exactly, but once you're sampling a significant proportion of the entire population (If you're doing non-destructive testing, anyway) you get a bonus to the standard error that's roughly the equivalent of multiplying by (N-n)/N.
However, this would only kick in once you have a really huge sample anyway, and the asymptote is still inspecting the entire population, so I'm not sure how much that helps you.

For more on this type of sampling for proportion defective, Deming wrote a lot about it in his books on sampling and statistics, as that was a big part of what he worked on.
(In particular, when doing destructive testing, depending on how lots are drawn, etc, you can have funny effects where discovering a large number of defective in the sample means there are fewer defective going to the customer.)
