Sample size calculation for precision at $k$: a/b-testing a recommender system I would like to conduct an online experiment to compare two different versions of a recommender system. The system returns a list of $r$ ranked recommendations. I would like to evaluate the recommender versions using the precision at $k$, where $k \leq r$, and $k$ is a small number, usually with $k \leq 10$. The precision at $k$ will be calculated as the share of the first $k$ recommendations which receive a click within the same web session. So if a users clicks on 2 of the first 5 items, the precision at 5 is 2/5 = 0.4. The individual unit of experimentation is the web session (and the recommendations don't change within a session), and we will average the precision at $k$ over all web sessions within each group to compare the two versions of the recommender.
My question is: How large a sample do I need?
For $k=1$, this could be calculated with standard formulas for comparing binomial proportions, because the item on the first position is either being clicked on or not, and the metric we look at equals the number of clicks, divided by the number of sessions. Of course, that needs the usual assumptions on parameters like significance level and sample size.
But how can the sample size be estimated for $k>1$?

It would be easy to assume here that you could just use the proportions for the control and treatment groups. Let's say you have 1,000 sessions in the control group and 1,000 sessions in the treatment group and you're looking at the first five recommendations for each. Then you have two proportions based on samples of 5,000.
In general, the problem with this approach is that you really don't have 5,000 independent samples. The dependence of samples (recommendations) within one session can be expressed in terms of the intraclass correlation coefficient. You'll need to estimate this and the adjust your sample size calculations accordingly. If you're using R, the package clusterPower should have a function to help you with these calculations, perhaps this one.
Per the example above, each sessions would constitute a cluster. So for the treatment group, you'd have 1,000 clusters with 5 observations in each cluster, etc.
