Interview Question: A/B Survey Results Non-Random Sampling How would you respond to the following case/question:
Assume you are testing the effect of an ad campaign at a social media company. The study is divided into two groups, 5,000 subjects who are shown an ad for why Coffee Shop X is better than Coffee Shop Y (group A), and 5,000 subjects who are not shown the ad (group B). The subjects are given a survey with a simple yes/no question of "do you find Coffee Shop X better than Coffee Shop Y?". Group A responds yes to the question 70% of the time, while group B responds yes to the question 65% of the time.
However, the samples are not representative of the overall population (assume for standard reasons why surveys are biased). But being a social media company with extensive data, you have access to thousands of features on demographics (i.e., age, sex, etc.) and usage characteristics (i.e., time spent on pages, subjects of interest, etc.) for both the samples and the overall population. What would be your framework for assessing whether the ad campaign really works (convinces more people to prefer Coffee Shop X than in the case of no ad)?
My initial inclination would be that you should build a predictive model with yes/no as the outcome and various features as the predictors on the sample data.


My initial inclination would be that you should build a predictive model with yes/no as the outcome and various features as the predictors on the sample data.

This would be useful in so far as creating a propensity score model,  but it is not sufficient in and of itself.  The problem is that building a predictive model would only tell you who is likely to say yes/no conditioned on the data you have, which were admittedly biased.
Even though the experiment is designed as a randomized control trial (or AB test), there is the problem of selection bias.  Namely, people self select to respond to surveys, and so the effect you estimate is not the effect of the ad campaign, it is the effect of the add campaign on those people who are the type of person to respond to surveys.  Very different.
This paper offers some insight into the problem.  Let $y$ be the outcome of interest, $x$ be a binary indicator for exposure, and $S$ be a binary indicator for entry into the data pool (i.e. data are collected if $S=1$ and are not otherwise).  In the introduction, the authors allude to a corollary they present in which they demonstrate that the conditional distribution $P(y\vert x)$ (which is what we want) is not recoverable if $S$ is a function of both the exposure $x$ and the outcome $y$.  So if being prompted about liking coffee shop X better than coffee shop Y and actually liking one shop over the other affect your probability of responding to the survey, then the problem can not be solved; we could not obtain an unbiased estimate of the treatment effect.  So, we would need to start with the assumption that this is not true.  From there, the linked paper presents some methods for possibly obtaining $P(y\vert x)$ from $P(y\vert x, S=1)$.  I would start there.
