What to do if you Double count in experimental design? I have to run A/B/n tests for a subscription service. Generally computing metrics for this situtation is ok:

*

*For example, coversion rate experiments. We have 1000 prospects in a group, and (say) 89 get through the entire funnel to become customers. So the empirical conversion rate is 89/1000

Now suppose we are in the following situation:
A customer can perform an action multiple times during the duration of an experiment. For example they can pause their subscription. Now events aren't unique.
Say we have two customers in an experiment, but one pauses twice during the experiment. We can't define the the pause rate as total_pauses/total_people as, in this case, the pause rate can be over 100%.
What do we do in this situation?

*

*Define rate metrics to be whether a customer did a reccuring action over total number of customers. So in our example we get a pause rate of 50%

*Switch to count metrics, where we report the number of times an action occurred per customer. So we'd get 1 pause per customer in our example (2 pauses/2 people).

Is there anything else? Would we need to modify our inference procedures in any way (e.g. use a different statistical test, other than a t-test)?
Do we have to modify the test we use to compute p-values when we double count? Or do we modify the things we count to stop double counting?

First define a metric that monitors the customer behavior you are interested and that is related to the business goal you want to optimize for. Then pick an appropriate statistical procedure.
Take your proposed count metric. What if 10% of customers are "indecisive" and pause two or more times, 10% of customer don't pause at all and the remaining 80% pause once. The count metric gives 1 pause per customer, on average. Is this a useful summary (of the hypothetical situation)? Or perhaps it's better to look at the probability that a customer doesn't pause at all? Statistics cannot tell you what's the best metric for your business/product.
