How do I calculate the uncertainty in process which estimates binomial distributions and has its own error rate We have a series of binomial distributions, each with separate and distinct chance of success k. For some small subset of these distributions we have an absolute measure of k from the full population.
We also have a process which can look at individual trials of the binomial distributions and estimate whether they succeeded or failed. This process has an error rate.
We only have a subset of the trials from each of the distributions but we can estimate k with our process, taking our sample, determining the proportion of success to failure.
What I'm unsure of is how to derive a measure of confidence in the output of this. For the distributions where we have the known k we can calculate the percentage point difference between what we calculate, but any differences could be due to either error in the success labelling process, or the fact that we're using a sample. Over a large enough number of distributions is it possible to separate out the error of the process?
I considered using fishers exact to test the likelihood that the sample would be drawn if the two distributions were the same, but that's not quite what I'm after.
Ideally I'd like to be able to construct some confidence interval for a given distribution. "Based on the size of our sample and some estimated error in the success/failure labelling process we expect the true value to lie within this range % of the time".
example results:




Estimated Sample Successes
Estimated Sample Failures
Population Successes
Population Failures




22
80
336
892


8
36
226
525


6
13
100
175


1
9
11
137




A number of confidence intervals are in use for the probability of success, which vary in the complexity of their computation and in their coverage probability $P_{cov}$ (which depends on the unknown true probability $p$).
The interval with $P_{cov}$ guaranteed to be greater than the nominal level $1-\alpha$ for all possible values of the true probability $p$ is known as the "Clopper-Pearson interval" or the "exact interval". In many cases, its coverage probability is considerably greater then $1-\alpha$, though. Therefore other intervals are in use that have $P_{cov}<1-\alpha$ for some values of $p$, but deviate not too much. Brown et al. (2001) found the "Wilson interval" to be a good compromise, which has the nice property of being computable through a closed formula. In my own experiments (second reference given below), I found the "Bayesian Highest Posterior Density interval" to have better coverage probability than the Wilson interval for true values $p$ close to one or zero (typical for estimating error/recognition rates), and this despite its smaller width. Its downside is that it is much more difficult to compute (no closed formula, but a computer algorithm).
The R package binom implements many confidence intervals (albeit not the Bayesian HPD interval) in the function binom.confint:
> library(binom)
> k <- 1
> n <- 10
> binom.confint(k,n, conf.level=0.95, methods=c("wilson","exact"))
  method x  n mean       lower     upper
1  exact 1 10  0.1 0.002528579 0.4450161
2 wilson 1 10  0.1 0.017876213 0.4041500

The HPD interval can be computed with the R package HDInterval as follows:
> library(HDInterval)
> k <- 1
> n <- 10
> hdi(qbeta, 0.95, shape1=(k+1), shape2=(n-k+1))
        lower     upper 
0.006301509 0.367513183


Brown, Cai, DasGupta: "Interval estimation for a binomial proportion." Statistical science, vol. 16, no. 2, pp. 101–117, 2001
Dalitz: "Construction of confidence intervals." Technical Report No. 2017-01, pp. 15-28, Hochschule Niederrhein, Fachbereich Elektrotechnik und Informatik, 2017

