  A sequence of recent papers has considered the role of measurement scales in
information retrieval (IR) experimentation, and presented the argument that
(only) uniform-step interval scales should be used, and hence that well-known
metrics such as reciprocal rank, expected reciprocal rank, normalized
discounted cumulative gain, and average precision, should be either discarded
as measurement tools, or adapted so that their metric values lie at
uniformly-spaced points on the number line. These papers paint a rather bleak
picture of past decades of IR evaluation, at odds with the community's overall
emphasis on practical experimentation and measurable improvement.
  Our purpose in this work is to challenge that position. In particular, we
argue that mappings from categorical and ordinal data to sets of points on the
number line are valid provided there is an external reason for each target
point to have been selected. We first consider the general role of measurement
scales, and of categorical, ordinal, interval, ratio, and absolute data
collections. In connection with the first two of those categories we also
provide examples of the knowledge that is captured and represented by numeric
mappings to the real number line. Focusing then on information retrieval, we
argue that document rankings are categorical data, and that the role of an
effectiveness metric is to provide a single value that represents the
usefulness to a user or population of users of any given ranking, with
usefulness able to be represented as a continuous variable on a ratio scale.
That is, we argue that current IR metrics are well-founded, and, moreover, that
those metrics are more meaningful in their current form than in the proposed
"intervalized" versions.

Information Retrieval