Data Science, Statistics, Analytics, Experiment, Causality.
conversion rate is a stable 20% and we get 1000 new visitors per week. How many weeks should we run the experiment, if our decision rule is to ship B if it’s significantly better than A? We can present this plot: Effect size of 20% is unrealistic. If we care about 10%, then a 6-week experiment