  Effective methodologies for evaluating recommender systems are critical, so
that such systems can be compared in a sound manner. A commonly overlooked
aspect of recommender system evaluation is the selection of the data splitting
strategy. In this paper, we both show that there is no standard splitting
strategy and that the selection of splitting strategy can have a strong impact
on the ranking of recommender systems. In particular, we perform experiments
comparing three common splitting strategies, examining their impact over seven
state-of-the-art recommendation models for two datasets. Our results
demonstrate that the splitting strategy employed is an important confounding
variable that can markedly alter the ranking of state-of-the-art systems,
making much of the currently published literature non-comparable, even when the
same dataset and metrics are used.

Information Retrieval