A/B testing segments post-analysis have the data of an A/B testing experiment. We can assume that the power analysis has been done correctly, so both sets, control and treatment had the necessary samples to find stat sig results.
The thing is that within the control/treatment sets, exist 3 segments, which I would like to analyze and detect if any result within each segment is stat sig, in this case, we can't assume that power analysis was done for each one of the segments.
How can I analyze the whole experiments and the sub-segments?
My idea until now is:
For the whole dataset: perform a stat sig test
For the segments: perform a stat sig test, but adjust the alpha to a lower value, kind of a Bonferoni adjustment.
Any ideas on other methods to evaluate the segments?

As you suspect, there's nothing you can do to increase the power of hypothesis tests on the segments after the fact. You're stuck with the power you have. The best thing you can do is control the Type 1 error in an efficient way so you don't lose power.
Remember that any time you perform a hypothesis test, you risk making a Type I error. In your suggested approach, you control the Type I error in the whole dataset and the segments separately. It is possible to incorrectly reject the null hypothesis on the whole data and incorrectly reject the null of at least one of the segments. In more technical terms, you are controlling the experimentwise error rate but not the family-wise error rate.
I have two suggestions.

*

*I would gate your hypothesis tests of the segments behind the test of the entire population. This means you first perform a hypothesis test on the entire population and only continue to perform tests on the segments if your test on the entire population is significant.


*The Bonferonni procedure is very inefficient. Several other procedures are more powerful and still maintain the same Type 1 error rate. I recommend the Hochberg Procedure. It is explained in the link I provided. The Hommel procedure is more powerful than Hochberg, but it is more complicated and fewer people are familiar with it. Both procedures can be implemented in R.
To summarize, perform a hypothesis test on the whole population, then, if you reject that hypothesis, move on to testing the segments using a Hochberg procedure.
