Basic Sampling - Provide confidence of estimate I tried searching the site but nothing came up.
I have a simple situation for a business problem - We have a population of 50 million files. We want to review a sample and see whether a file contains or does not contain "x". So it's binary. We will calculate a proportion of yes:no.
We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way. Rather, confidence intervals are supposed to be interpreted as "in the long-run, if we conduct many experiments, we are 95% confident that the CIs calculated for those samples will contain the true population parameter". Do we just take an arbitrary "large enough" random sample, calculate the proportion and assume that is the true proportion? I came across the binomial credible interval that can be interpreted as the former, but I know nothing of Bayesian inference unfortunately.


We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way.

You are correct. Our confidence (or belief) is undefined unless we start with a Bayesian prior belief. In that case, we can update to a posterior belief based on the experiment. But the posterior will depend on the prior. For example, suppose our prior belief is the proportion is about 50%. And suppose our confidence interval is [30%, 40%]. We still might not believe the true parameter is in that low range. Instead we believe the experiment is a fluke.
The confidence-interval approach avoids making any claims about what we should believe post-hoc. However, it does give a useful guarantee: no matter what the true proportion is, with 95% probability (or 99.9%, etc), it will be in the confidence interval. In other words, if you make a claim "the true proportion is in this interval", that claim will only be wrong with probability 5% (or 0.1%, etc).
But when you see the actual interval resulting from your experiment, you might use prior knowledge to decide whether the 5% event (or 0.1%, etc) happened or not. That would be a Bayesian approach as I mentioned above.
