Dealing with pre-test bias by repeated random sampling I'm dealing with zero inflated data that has extreme volatility. I wish to randomly assign X observations/participants/subjects from the data, treat half of them with a new treatment ("target") while not treating the other half ("base"). I other words, I wish to conduct an A/B test.
The problem is that even if I randomly choose 200K observations to each group (base and target), I still see a pretty big difference between them before the treatment - a "pre-test bias". This difference can get up to 4%, which is usually more than the impact of my actual treatment.
My aim is to reduce this pre-test bias as much as I can, without hindering the validity of my tests.
This is what I have in mind:

*

*Randomly sample X observations from my population that will be my sample for the experiment.

*Randomly divide the sample to "base" and "target".

*Calculate the initial difference between base and target in %, before doing anything (the pre-test bias).

*If the difference is bigger than 1% (to any direction), repeat phases 2 and 3.

*Only when I calculate a small enough difference, conduct my experiment.

Is this a valid method?
What are the disadvantages? Do you recon it has any advantages against a balanced sampling method such as the "cube method"? Some of my colleagues have concerns regarding the randomity of my suggestion, but we are unable to reach a verdict.
Many thanks!
Matan

This is a topic that comes up a lot in randomized clinical trials. As a result, there's an extensive literature on covariate adaptive randomization in that area (see e.g. this review). However, how you randomize affects how you should analyze ("Analyze as you randomize."), see e.g. this paper and this one, which especially matters if you do anything particularly complicated. E.g. I'm not, at all, sure how to handle your randomization scheme in the analysis. Some simple schemes that make the analysis straightforward:

*

*Randomize by strata with separate strata for those without a previous event and those with a previous event. This ensure those two groups will be exactly balanced within the base and target arms, the main question is whether you'd still be concerned about possible imbalances amongst those with previous events (for some possible strategies to address see further down).

*Randomize each record sequentially in such a way that it is more likely that balance will be improved. Example:

*

*There's too many zeros in the base arm and too few in the target arm, another zero record comes up, we assign it to the target arm with probability 0.75 and to the base arm with probability 0.25 (and the other way around if the imbalance is the other way, 1:1 if there's no imbalance). Or any other scheme that's similar and you just go through the records one after another trying to balance each time.

*Under that scheme, you record the probability each item had to be assigned to each arm and then adjust for that in the analysis.

*If you just have a few possible assignment options (e.g. 0.25:0.75, 0.5:0.5, 0.75:0.25), then we can just do the analysis stratified by assignment stratum.

*Note that this approach assumes that your observations are independent, if that's not the case (e.g. with infectious diseases intervening on one household member, for example with a vaccine, might substantially affect the infection probability of other household members), things become a lot more complicated. One easy way to avoid complications could be to randomize at a level where indepence exists again (e.g. in the infectious disease example at the household or community level).

*Importantly, you never want there to be too high a probability to be assigned to one arm or the other (otherwise as the probability for one arm goes towards 1.0, the record no longer contributes information for the comparison of interest).



The other question is whether you can cope with a certain level of imbalance. How much does this really affect your analysis, especially if you can use the pre-intervention levels as a covariate in your analysis model (usually an obvious thing to do)? Are there transformations (e.g. with some variables like the albumin-creatinine ratio in patients, because regression residuals are really not normal so that you'd probably want to log-transform to not have one or two really extreme values massively drag the arithemtic mean upwards by chance) or models (e.g. for count data a negative binomial or zero-inflated negative binomial model with adjustment for the logarithm of 0.5 + the number of previous events) that are less affected by extreme values? You might want to do these things, even if you balance covariates via your randomization (but check the literature whether this impacts how you do it, e.g. I'm not 100% sure whether you should have different covariate coefficients per stratum).
Also note, that in a randomized experiment the goal of the randomization is not per-se to balance the groups at baseline (see e.g. the extensive discussion here with links), but to ensure you understand the randomness process. That does not mean you'd not want to increase the efficiency of your analysis by e.g. adjusting for known prognostic variables in your analysis (or perhaps balancing covariates also helps).
