title: Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
    Prediction
summary: Large Language Models (LLMs) have demonstrated exceptional capabilities in
  generalizing to new tasks in a zero-shot or few-shot manner. However, the
  extent to which LLMs can comprehend user preferences based on their previous
  behavior remains an emerging and still unclear research question.
  Traditionally, Collaborative Filtering (CF) has been the most effective method
  for these tasks, predominantly relying on the extensive volume of rating data.
  In contrast, LLMs typically demand considerably less data while maintaining an
  exhaustive world knowledge about each item, such as movies or products. In this
  paper, we conduct a thorough examination of both CF and LLMs within the classic
  task of user rating prediction, which involves predicting a user's rating for a
  candidate item based on their past ratings. We investigate various LLMs in
  different sizes, ranging from 250M to 540B parameters and evaluate their
  performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct
  comprehensive analysis to compare between LLMs and strong CF methods, and find
  that zero-shot LLMs lag behind traditional recommender models that have the
  access to user interaction data, indicating the importance of user interaction
  data. However, through fine-tuning, LLMs achieve comparable or even better
  performance with only a small fraction of the training data, demonstrating
  their potential through data efficiency.
paper_authors._id: 645da55c680734460f9fbfbd
paper_authors.name: Wang-Cheng Kang
paper_authors._id: 645da55c680734460f9fbfbe
paper_authors.name: Jianmo Ni
paper_authors._id: 645da55c680734460f9fbfbf
paper_authors.name: Nikhil Mehta
paper_authors._id: 645da55c680734460f9fbfc0
paper_authors.name: Maheswaran Sathiamoorthy
paper_authors._id: 645da55c680734460f9fbfc1
paper_authors.name: Lichan Hong
paper_authors._id: 645da55c680734460f9fbfc2
paper_authors.name: Ed Chi
paper_authors._id: 645da55c680734460f9fbfc3
paper_authors.name: Derek Zhiyuan Cheng
paper_title: Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
    Prediction
paper_summary: Large Language Models (LLMs) have demonstrated exceptional capabilities in
  generalizing to new tasks in a zero-shot or few-shot manner. However, the
  extent to which LLMs can comprehend user preferences based on their previous
  behavior remains an emerging and still unclear research question.
  Traditionally, Collaborative Filtering (CF) has been the most effective method
  for these tasks, predominantly relying on the extensive volume of rating data.
  In contrast, LLMs typically demand considerably less data while maintaining an
  exhaustive world knowledge about each item, such as movies or products. In this
  paper, we conduct a thorough examination of both CF and LLMs within the classic
  task of user rating prediction, which involves predicting a user's rating for a
  candidate item based on their past ratings. We investigate various LLMs in
  different sizes, ranging from 250M to 540B parameters and evaluate their
  performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct
  comprehensive analysis to compare between LLMs and strong CF methods, and find
  that zero-shot LLMs lag behind traditional recommender models that have the
  access to user interaction data, indicating the importance of user interaction
  data. However, through fine-tuning, LLMs achieve comparable or even better
  performance with only a small fraction of the training data, demonstrating
  their potential through data efficiency.
paper_ai_summary: LLMs show data-efficient potential in user rating prediction through fine-tuning, though they lag behind CF methods in zero-shot scenarios.
paper_ai_keywords: Large Language Models (LLMs) Collaborative Filtering (CF) user rating prediction zero-shot few-shot fine-tuning data-efficient