Interview Question: A/B Survey Results Non-Random Sampling How would you respond to the following case/question:
Assume you are testing the effect of an ad campaign at a social media company. The study is divided into two groups, 5,000 subjects who are shown an ad for why Coffee Shop X is better than Coffee Shop Y (group A), and 5,000 subjects who are not shown the ad (group B). The subjects are given a survey with a simple yes/no question of "do you find Coffee Shop X better than Coffee Shop Y?". Group A responds yes to the question 70% of the time, while group B responds yes to the question 65% of the time.
However, the samples are not representative of the overall population (assume for standard reasons why surveys are biased). But being a social media company with extensive data, you have access to thousands of features on demographics (i.e., age, sex, etc.) and usage characteristics (i.e., time spent on pages, subjects of interest, etc.) for both the samples and the overall population. What would be your framework for assessing whether the ad campaign really works (convinces more people to prefer Coffee Shop X than in the case of no ad)?
My initial inclination would be that you should build a predictive model with yes/no as the outcome and various features as the predictors on the sample data.

Because this is an interview question, I'll structure my response in formats that are popular in case study circles:

*

*First, a clarifying question. Can we trust the survey? That is, was the survey designed and performed with concerns for non-response, biasedness in mind?

*

*I'll assume we can trust it and the headline result. This gives us a good starting point for our analyses.



*Next, I'd want to see if the people we care about reaching are in both groups. Who responded to the surveys? Does it make sense to compare these groups, or do we have apples and oranges? To do this, I'll compare summary statistics on some set of observed baseline characteristics I hypothesize are relevant to preferring cafes (demographics, region, etc.).

*If the groups are not too different baseline, I'll proceed with a selection-on-observables strategy and estimate if there's a meaningful difference in response by group. A popular way to do this is by assigning weights to the individuals via propensity score matching and then estimating a regression of the outcome on the treatment indicator (group A or B). The results from this will help me give a rigorous answer to the question. (There's careful work to be done for calculating the right standard error for the treatment estimate, but I'll skip details here in an interview)

*Finally, I'd present results from the survey and my analysis to the stakeholder and have them weigh in on the approach.

