Retention Induced Biases in a Recommendation System with Heterogeneous Users
<|reference_start|>Retention Induced Biases in a Recommendation System with Heterogeneous Users: I examine a conceptual model of a recommendation system (RS) with user inflow and churn dynamics. When inflow and churn balance out, the user distribution reaches a steady state. Changing the recommendation algorithm alters the steady state and creates a transition period. During this period, the RS behaves differently from its new steady state. In particular, A/B experiment metrics obtained in transition periods are biased indicators of the RS's long term performance. Scholars and practitioners, however, often conduct A/B tests shortly after introducing new algorithms to validate their effectiveness. This A/B experiment paradigm, widely regarded as the gold standard for assessing RS improvements, may consequently yield false conclusions. I also briefly discuss the data bias caused by the user retention dynamics.<|reference_end|>
@article{ma2024retention,
    title={Retention Induced Biases in a Recommendation System with Heterogeneous
  Users},
    author={Shichao Ma},
    journal={arXiv preprint arXiv:2402.13959},
    year={2024},
    doi={10.1007/978-3-031-71975-2_2},
    archivePrefix={arXiv},
    eprint={2402.13959},
    primaryClass={cs.IR}
}