Bootstrap in cluster experiments I am planning an AB-test for something like a call-center. We are testing a new interface for the call-center operators. It has to be tested within a small group of roughly 20 operators. The main metric is the average call processing time (CPT). The experiment is supposed to continue for 1 week, each operator will process about 2000 calls during that time.
The operators are considered permanently fixed, all of them have a history of a few weeks and they are supposed to continue working during the AB-test period.
Suppose we take a single random split of the group: 10 into the control group, 10 into the test group.
What would be the right bootstrap procedure to calculate the p-value of the test?
It seems that we should not sample the operators themselves. We can sample tickets for each operator (the number of tickets that we expect him to process during the experiment), calculate the between-groups CPT delta and get the basic bootstrap distribution, to be used for the p-value calculation later. Seems right so far?
But, if we just take a single random split of the group and observe the average group CPT in the past weeks, we will see that the lines are very different. The operators are very different, and the CPT distributions are long-tailed.
The accepted way to handle this seems to be to go over different splits and find the best one, which gives the best similarity between the CPT dynamics of the groups in the past.
But it is pretty obvious that this split is very special, and it cannot be considered just a single random split.
Can we still use the basic bootstrap approach in this case?
If not, how can we correct for the overfitting that we have probably done by selecting this very special group split?

With clustered data, as you've described, you'll want to bootstrap along the lines of the clustering. I would say you do want to bootstrap operators first. Then within each new bootstrapped operator, you bootstrap their calls (this assumes that calls are independent - i.e., the same person doesn't call twice, which may or may not be the case). So for example, in one run of bootstrapping operator #2 could appear three times, each with a different bootstrapped sample of their calls. See for example this paper (section 2.1 and Figure 1 describe the general procedure).
