  We consider the problem of retrieving and ranking items in an eCommerce
catalog, often called SKUs, in order of relevance to a user-issued query. The
input data for the ranking are the texts of the queries and textual fields of
the SKUs indexed in the catalog. We review the ways in which this problem both
resembles and differs from the problems of IR in the context of web search. The
differences between the product-search problem and the IR problem of web search
necessitate a different approach in terms of both models and datasets. We first
review the recent state-of-the-art models for web search IR, distinguishing
between two distinct types of model which we call the distributed type and the
local-interaction type. The different types of relevance models developed for
IR have complementary advantages and disadvantages when applied to eCommerce
product search. Further, we explain why the conventional methods for dataset
construction employed in the IR literature fail to produce data which suffices
for training or evaluation of models for eCommerce product search. We explain
how our own approach, applying task modeling techniques to the click-through
logs of an eCommerce site, enables the construction of a large-scale dataset
for training and robust benchmarking of relevance models. Our experiments
consist of applying several of the models from the IR literature to our own
dataset. Empirically, we have established that, when applied to our dataset,
certain models of local-interaction type reduce ranking errors by one-third
compared to the baseline tf-idf. Applied to our dataset, the distributed models
fail to outperform the baseline. As a basis for a deployed system, the
distributed models have several advantages, computationally, over the
local-interaction models. This motivates an ongoing program of work, which we
outline at the conclusion of the paper.

Information Retrieval