Estimate sample size from a variable population Context:
I want to measure accuracy, precision, and recall for individual raters. Each rater completes a variable amount of labels, for ex. rater A may complete 500 in a given time period while rater B may complete 800.
I want to sample a fixed number of labels from each rater to measure their performance.
My confusion: if each rater is completing a variable amount of labels and each rater has a different skill level, how do I determine the sample size needed to test each rater? For simplicity, I'd like to use a fixed sample size, understanding that it may provide a different level of confidence for each rater (right?)

For assessing accuracy, each result is accurate or not, so you have a binomial process. The confidence level for a given sample size is well known, see e.g. https://reliabilityanalyticstoolkit.appspot.com/sample_size. You can use the same size of sample for each rater, and you will get the same confidence interval for each. (With a sample of 500 it will be disappointingly broad - and to halve the confidence interval, you need a sample four times the size.)
For precision, the situation is similar, but your sample consists only of true positives that they have reviewed.
For recall, surely the work rate matters: someone accurate but slow will have a low recall.
