What is the best way to evaluate the efficacy of an intervention that only a non-random subset of the participants complies with? I want to evaluate the efficacy of an intervention that only about 50% of participants comply with. The intervention consists of giving people an app that they are asked to use over an extended period of time. The problem is that only about 50% of the participants who are asked to use the app do so consistently. We have reasons to believe that the people who comply with the intervention may be systematically different from those who do not. The outcome measures are self-report measures that might suffer from placebo effects or demand characteristics. Our question is how beneficial the app is for people who actually use it.
I can choose both the experimental design and the statistical methods used to analyze the data. What is the best combination of an experimental design and data analysis strategy?

Assuming that you are randomising participants to use/not use the app, and that’s all you are randomising (i.e. you are not also giving them $1000 when you assign them to a condition), a much simpler approach is to treat the assignment as an instrumental variable for the treatment of actually using/not using the app. You will be estimating something called the local average treatment effect, which is the effect of app use among people who use the app because they were assigned to use the app. No need to adjust for any common causes of app use/the outcome.
