Search engines typically output search items in a ranked manner, where a search item that is determined to be highly relevant to an input query and/or user is displayed relatively high on a list of search items when compared to a search item that is determined to be less relevant to the query and/or user. Properly ranking search items is an important task, as a typical user is not willing to sift through several pages of search items, but instead only reviews a first, relatively small number of search items. Thus, often a user will assume that a small subset of search items shown on a first page is most relevant to the user and query, when, in actuality, the user may have found a search item on a subsequent page that is most relevant to the user and query.
Ranking of search items is performed by ranking algorithms (rankers), which assign scores to search items that are located in response to a query. A higher score correlates to a higher position on a list of search items provided to a user in response to the query. Information retrieval metric methods are used to determine the quality of a ranking generated by a ranker. More specifically, in order to evaluate the quality of a ranker, that ranker is provided with labeled data (e.g., the relevance of search results to a query is known a priori) and outputs an ordered list of search items. An information retrieval metric method is then used to determine a quality of the rankers based upon the ordered list of search items. Furthermore, it has been determined that rankers may, in some instances, be combined and may provide better ranking scores when combined as compared to ranking scores output by the rankers individually. Determining how to combine rankers in a way that is optimal or near optimal for an information retrieval metric method or methods, however, is non-trivial.