A/B testing segments post-analysis have the data of an A/B testing experiment. We can assume that the power analysis has been done correctly, so both sets, control and treatment had the necessary samples to find stat sig results.
The thing is that within the control/treatment sets, exist 3 segments, which I would like to analyze and detect if any result within each segment is stat sig, in this case, we can't assume that power analysis was done for each one of the segments.
How can I analyze the whole experiments and the sub-segments?
My idea until now is:
For the whole dataset: perform a stat sig test
For the segments: perform a stat sig test, but adjust the alpha to a lower value, kind of a Bonferoni adjustment.
Any ideas on other methods to evaluate the segments?

Assuming the outcome is binary (though the following will work for any outcome), we can:

*

*Model an interaction between treatment and segment, and

*Test for an effect of the treatment through a likelihood ratio test.

Let's assume we have designed this experiment with the hypothesis that there is an effect and it varies by group.  Doing otherwise (looking at the data first and then forming a hypothesis) would be an invalid approach. Assume we run the experiment and our data look like
    txt segment     y     N
  <int> <fct>   <int> <int>
1     0 1         143  1669
2     0 2         374  1596
3     0 3         485  1735
4     1 1         211  1701
5     1 2         645  1673
6     1 3         241  1626

Using R, we can fit a model which allows treatment effect to vary by group using
model = glm(cbind(y, N-y) ~ txt*segment, data = d, family = binomial())

The summary of the model can be examined to determine the estimated effects of each group and treatment.  I find it easier to plot the predicted probabilities.  For the data I presented, such a plot would look like...

It certainly looks like there is an effect of treatment and that effect varies by group. Let's test that with a likelihood ratio test.  First, we create a null model which adjusts for group in an additive fashion.  Then, we pass our model and our null model to the anova function.
# Create a null model where there is no effect of group
null_txt_model = glm(cbind(y, N-y) ~ txt + segment, data = d, family = binomial())
anova(null_txt_model, model, test='LRT')

Analysis of Deviance Table

Model 1: cbind(y, N - y) ~ txt + segment
Model 2: cbind(y, N - y) ~ txt * segment
  Resid. Df Resid. Dev Df Deviance  Pr(>Chi)    
1         2     182.91                          
2         0       0.00  2   182.91 < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Our model leads to a reduction in deviance large enough to reject the null hypothesis.  We would conclude that the treatment effect does depend on group.  Had we failed to reject this, we could not rule out a purely additive effect of group.
The problem with this analysis is that if you've powered your experiment to detect an effect of the treatment, you've likely underpowered it for this sort of analysis.  Nothing you can do about it at this point.
