Recommendation systems that provide appropriate solutions to users to reduce their decision complexity have become popular in the Internet world. Designing and evaluating such systems remain essential challenges to researchers and practitioners. Toward that end, a critical task is how to obtain user preferences. Mobile phones have become indispensable in everyday life, yet fierce market competition, characterized by rapid introductions of different models with novel designs and advanced features, have made consumers’ purchase decision making increasingly complex. As a well-established, multiple criteria decision technique, analytic hierarchy processing (AHP) provides an intuitive model of a hierarchical structure capable of supporting complex product comparisons and evaluations by consumers. In this paper, we illustrate the application of an AHP-based mechanism to develop a Web-based recommendation system and empirically evaluate the prototype by conducting a controlled experiment with 244 mobile phone users, focusing on both content and system satisfaction. Our evaluation includes benchmark systems built on rank-based analysis and an equal weight-based system as comparative baselines. Overall, the results suggest the viability and value of using AHP to construct effective recommendation systems. Subjects appear satisfied with the recommendations by the AHP-based system, though its relatively demanding input requirements may need mitigation and adequate interface designs. This study contributes to research and practice in recommender systems in general and helps develop mobile phone recommendation systems for online stores and consumers in particular.