Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
  Prediction
Large Language Models (LLMs) have demonstrated exceptional capabilities in
generalizing to new tasks in a zero-shot or few-shot manner. However, the
extent to which LLMs can comprehend user preferences based on their previous
behavior remains an emerging and still unclear research question.
Traditionally, Collaborative Filtering (CF) has been the most effective method
for these tasks, predominantly relying on the extensive volume of rating data.
In contrast, LLMs typically demand considerably less data while maintaining an
exhaustive world knowledge about each item, such as movies or products. In this
paper, we conduct a thorough examination of both CF and LLMs within the classic
task of user rating prediction, which involves predicting a user's rating for a
candidate item based on their past ratings. We investigate various LLMs in
different sizes, ranging from 250M to 540B parameters and evaluate their
performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct
comprehensive analysis to compare between LLMs and strong CF methods, and find
that zero-shot LLMs lag behind traditional recommender models that have the
access to user interaction data, indicating the importance of user interaction
data. However, through fine-tuning, LLMs achieve comparable or even better
performance with only a small fraction of the training data, demonstrating
their potential through data efficiency.
645da55c680734460f9fbfbd
Wang-Cheng Kang
645da55c680734460f9fbfbe
Jianmo Ni
645da55c680734460f9fbfbf
Nikhil Mehta
645da55c680734460f9fbfc0
Maheswaran Sathiamoorthy
645da55c680734460f9fbfc1
Lichan Hong
645da55c680734460f9fbfc2
Ed Chi
645da55c680734460f9fbfc3
Derek Zhiyuan Cheng
Do LLMs Understand User Preferences? Evaluating LLMs On User Rating
  Prediction
Large Language Models (LLMs) have demonstrated exceptional capabilities in
generalizing to new tasks in a zero-shot or few-shot manner. However, the
extent to which LLMs can comprehend user preferences based on their previous
behavior remains an emerging and still unclear research question.
Traditionally, Collaborative Filtering (CF) has been the most effective method
for these tasks, predominantly relying on the extensive volume of rating data.
In contrast, LLMs typically demand considerably less data while maintaining an
exhaustive world knowledge about each item, such as movies or products. In this
paper, we conduct a thorough examination of both CF and LLMs within the classic
task of user rating prediction, which involves predicting a user's rating for a
candidate item based on their past ratings. We investigate various LLMs in
different sizes, ranging from 250M to 540B parameters and evaluate their
performance in zero-shot, few-shot, and fine-tuning scenarios. We conduct
comprehensive analysis to compare between LLMs and strong CF methods, and find
that zero-shot LLMs lag behind traditional recommender models that have the
access to user interaction data, indicating the importance of user interaction
data. However, through fine-tuning, LLMs achieve comparable or even better
performance with only a small fraction of the training data, demonstrating
their potential through data efficiency.
LLMs show data-efficient potential in user rating prediction through fine-tuning, though they lag behind CF methods in zero-shot scenarios.
Large Language Models (LLMs) Collaborative Filtering (CF) user rating prediction zero-shot few-shot fine-tuning data-efficient