What is the difference CUPED and regression adjustment？ CUPED (Controlled-experiment Using Pre-Existing Data) is a variance reduction technique created by Microsoft in 2013 and widely used in technology companies. let Y is the target metrics and X is pretreatment covariate, T is treat variable(to simplify the discussion, we assume that there are only two treatment group, T = 0 is control group, T = 1 is treat group)，the main idea of CUPED is:

*

*compute $\theta$
$$\theta = \frac{cov(Y, X)}{var(X)} = corr(X, Y) * var(Y) = \rho * var(Y)$$

*

*compute adjusted $Y_i^{cv} = Y_i - (X_i - \mu_X) * \theta$ for each user


*evaluate the AB test using $Y_i^{cv}$ instead of $Y_i$
the result cuped-adjusted estimate treat effect is
$$\tau = (\overline Y_1 - \theta * (\overline X_1 - \mu_X)) - (\overline Y_0 - \theta * (\overline X_0 - \mu_X)) \\= (\overline Y_1 - \overline Y_0) - \theta * (\overline X_1 - \overline X_0)$$
Another widely used and long-established method for increase power and adjusting preexisting differences is ANCOVA(Analysis of covariance) or regression-adjustment.
The ANCOVA model assumes a linear relationship between the response (Y) and covariate (X):
$$ Y = b_0 + \tau * T + \theta* X $$
the result regression-adjusted estimate treat effect is same as above, the only difference seems to be how to estimate $\theta$, so My questions is which one is more reasonable ?

For variance-reduction of randomized experiments/AB tests, ancova/regression-adjust is a more appropriate method to estimate $\theta$, because it takes into account the differences in pre-treat variables between treat/control group before experiment, and the treatment effect during experiment.
sim
cuped
Some simulation data are generated below:
library(tidyverse)
library(mvtnorm)

N <- 10000 # sample size
tau <- 1.2 # treat effect, to make the problem clearer, we assume a relatively large value
cor_mat <- matrix(c(1.0, 0.6, 0.6, 1.0), 2, 2) # assuming a moderately correlation between x and y
cor_mat

     [,1] [,2]
[1,]  1.0  0.6
[2,]  0.6  1.0

xy <- mvtnorm::rmvnorm(N, c(0.0, 0.0), cor_mat) 
df <- tibble::tibble(
  x = xy[, 1],
  t = sample(0:1, N, TRUE),
  y = xy[, 2] + t * tau
)

cuped suggests to estimate $\theta$ by from the pooled population of control and treatment. The way to manually calculate $\theta$ is as follows:
df %>%
  mutate(
    myg = mean(y), # global mean y
    mxg = mean(x), # global mean x
  ) %>%
  summarise(
    vary = sum((y - myg) ^ 2),
    varx = sum((x - mxg) ^ 2),
    covxy = sum((x - mxg) * (y - myg)),
    r = covxy / sqrt(vary * varx),
    b = covxy / varx
  )

the estimated correlation coefficient(r = 0.518) is smaller than the specified value (r = 0.6) due to simply pool strategy.
ancova/regression-adjust
ancova compute the y and x correlation separately for each group and find the pooled within-group correlation. This is also exactly the coefficient of the covariate estimated by ols
df %>%
  group_by(t) %>%
  mutate(
    myg = mean(y), # mean y for each group
    mxg = mean(x), # mean x for each group
  ) %>%
  ungroup() %>%
  summarise(
    vary = sum((y - myg) ^ 2),
    varx = sum((x - mxg) ^ 2),
    covxy = sum((x - mxg) * (y - myg)),
    r = covxy / sqrt(vary * varx),
    b = covxy / varx ## same as lm(y ~ t + x, df)$coef['x']
  )

the estimated correlation coefficient(r = 0.606) is close to the specified value (r = 0.6) .

