Does online A/B experimentation with simple randomisation ensure properties of a “probability sample”? High Level Problem Statement:
Traditionally vendors (Optimizely, LaunchDarkly etc), who provide in-product experimentation solution to run online A/B experiment, randomise in-coming active users of the product using a simple randomisation. Does this simple randomisation ensure properties of a “probability sample”?
We are concerned if this can be argued as convenience sampling (despite randomisation). What implicit and explicit assumptions do we have here or need to argue so that this is a probability sample and NOT a convenience sample?
If this is convenience sampling, I wonder how does people (Microsoft and other companies) do traditional A/B tests using t/z-tests. A lot of vendors allow A/B experimentation and tooling for example: Optimizely, Launch Darkly etc. I wonder what is happening here.
More details:
In product A/B randomised controlled experiments is executed in the following way. Usually we develop a feature and roll it out to (lets say) to 50% of users coming to the platform. The randomisation is done at the user level, when the users arrives at the product (details below).
However, as you can understand typically for designing an experiment we do pre-experiment power analysis for determining sample size and experimentation window. Now does this process make the sampling process a convenience sample?
How does the randomisation work?
We use hashing based mechanics to randomise the user when we setup the feature for release using A/B test. So randomisation scheme is pre-defined when setting up the experiment using the hash(user+experiment name + config). Therefore, when user comes we use the randomisation scheme to randomise. An example method is shown here.

It may be a convenience sample, only in so far as you can only collect data from people who choose to interact with your product during that time and not anyone else.
That being said, it's all about interpreting the inference correctly.  Is the result of the test an inference to how the change would effect all possible users?  No.  Its an inference to how the change would effect users who have been and will continue to use the product.
The randomization allows for a valid inference assuming a couple of things:

*

*That you interpret the population from which you are sampling correctly.  As you mention, there may be some hairs to split since you are not randomly sampling your user base and instead are relying on people to come to you.


*Assuming that the test statistic has the hypothesized distribution.  As an example, when the outcome is binary, the test large enough, and the baseline metric not too close to 0 or 1, then an inference via the z test is fairly good.


*Your randomization is working and not systematically introducing confounding.
The randomization ensures that the distribution of outcomes prior to intervention is identical, thus allowing for a causal inference to be made.
