  Offline evaluation of information retrieval and recommendation has
traditionally focused on distilling the quality of a ranking into a scalar
metric such as average precision or normalized discounted cumulative gain. We
can use this metric to compare the performance of multiple systems for the same
request. Although evaluation metrics provide a convenient summary of system
performance, they also collapse subtle differences across users into a single
number and can carry assumptions about user behavior and utility not supported
across retrieval scenarios. We propose recall-paired preference (RPP), a
metric-free evaluation method based on directly computing a preference between
ranked lists. RPP simulates multiple user subpopulations per query and compares
systems across these pseudo-populations. Our results across multiple search and
recommendation tasks demonstrate that RPP substantially improves discriminative
power while correlating well with existing metrics and being equally robust to
incomplete data.

Information Retrieval