Sample size calculation to bound CI around a true value I'm working on a cold start system for a recommender system - Whenever I have a new item, I need to show it to users enough times to get a somewhat accurate estimate of CTR, and then I will let the ML model score these items once CTR is estimated.
I am trying to find the minimum number of times I need to show the item before my observed CTR is close enough to the true CTR.
For example, would it be possible to estimate a number of shows needed to get a 95% CI on the CTR thats between some bound e.g. 0.1 of the true value?


I am trying to find the minimum number of times I need to show the item before my observed CTR is close enough to the true CTR.

You can only put probabilistic bounds on how far your estimate will be from the truth.
Sounds like you're interested in determining $n$, the number of trials, so that you bound the probability that your estimate is within some radius of the true value.  Using your values
$$   P(\vert \theta - \hat{\theta} \vert > 0.1) \leq p_\min$$
Where $\theta$ is the CTR and $\hat{\theta} = x/n$.  Here, $x$ are clicks and $p_\min$ is your desired probability threshold.
I asked a similar question on math exchange about 5 or 6 years ago. In that question, the answerer argues that if the sampling distribution of $\hat{\theta}$ is approximately normal, then we express $p_\min$ in terms of the sample size
$$ P( \mid \hat{\theta} - \theta \mid \geq \epsilon) \approx 2\mathbf{\Phi}\left( \dfrac{- \epsilon\sqrt{n}}{\sigma} \right)  $$
Because you're modelling a binary outcome, $\sigma \leq 0.5$ so we can just use $\sigma=0.5$ as an approximation.  Since $n$ is an integer, we just need to find an $n$ large enough so that this bound is tolerable.
You can see how good the bound does by just simulating a few scenarios. Here is a plot for various epsilon, theta, and sample size.

As you may be able to see, the bound os overkill when theta is small (top row).  The computed probability from the equation above (solid line) is always above the estimated probability achieved through simulation (dashed line).  The bound is most accurate when $\theta=0.5$ (bottom row) which makes sense for a few reasons (namely $\sigma=0.5$ in that case, and the sampling distribution of the mean is most normal).
