Bernoulli parameter estimation after every sample received I have a pretty simply-looking question on the parameter estimation. hope you can help me.
I have a biased coin and I want to decide if the probability of head $p$ is bigger or smaller than  known probability $p_0$ (and this I want to guess correctly with high probability, say  99%).
If I draw say $100$ samples, it is easy to compute the confidence intervals and decide or abstain. However, I don't want to decide on the number of samples beforhand, but rather want to draw just enough samples to be able to decide if $p > p_0$ or not. I can compute the confidence intervals after every sample, however it is not correct because the "prefixes" of sampled sequence are not independednt, thus, my predetermined confidence level is "poisoned".
The motivation is quite natural. If I toss the coin $100$ times and receive $100$ heads, then I can guarantee that $p > 0.5$ (with failure probability 1%) and it makes no sense to draw further. On the other hand, if $p \approx p_0$, I may need a lot more samples.
Unfortunatelly, I don't think there is a perfect universal answer, so I'm happy for any partial thougths.

Can't you approach this in a Bayesian way?
Place a beta prior on $p$, and then toss the coin repeatedly. After each toss, the posterior is updated and you check the amount of posterior probability for values above $p_0$.
Here is some rough code. I am assuming that p > p0, and using p0 = 0.5.
# reference value
p0 <- 0.5

# biased prob of head
p <- runif(n = 1L, min = p0, max = 1)

# threshold on posterior prob that p > p0
prob_threshold <- 0.99

heads <- 1L
tails <- 1L

for(i in 1:100){
  # new coin toss
  x <- rbinom(n = 1L, size = 1L, prob = p)

  # update counts
  if(x == 0L){
    tails <- tails + 1L
  }else{
    heads <- heads + 1L
  }

  # quantify the posterior prob above the reference value
  z <- pbeta(p0, shape1 = heads, shape2 = tails, lower.tail = FALSE)

  if(z >= prob_threshold){
    break
  }
}

As @whuber points out, you probably want some additional termination criterion. Here, I run at most 100 trials for example.
