document: Learning consensus opinion : mining data from a labeling game We consider the problem of identifying the consensus ranking for the results of a query , given preferences among those results from a set of individual users . Once consensus rankings are identified for a set of queries , these rankings can serve for both evaluation and training of retrieval and learning systems . We present a novel approach to collecting the individual user preferences over image-search results : we use a collaborative game in which players are rewarded for agreeing on which image result is best for a query . Our approach is distinct from other labeling games because we are able to elicit directly the preferences of interest with respect to image queries extracted from query logs . As a source of relevance judgments , this data provides a useful complement to click data . Furthermore , the data is free of positional biases and is collected by the game without the risk of frustrating users with non-relevant results ; this risk is prevalent in standard mechanisms for debiasing clicks . We describe data collected over 34 days from a deployed version of this game that amounts to about 18 million expressed preferences between pairs . Finally , we present several approaches to modeling this data in order to extract the consensus rankings from the preferences and better sort the search results for targeted queries .
doc_bio_tags: O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O