title: FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in
    LLMs Elicits Effective Personalization to Real Users
summary: Effective personalization of LLMs is critical for a broad range of
  user-interfacing applications such as virtual assistants and content curation.
  Inspired by the strong in-context learning capabilities of LLMs, we propose
  Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a
  meta-learning problem. Under this framework, an LLM learns to quickly adapt to
  a user via a few labeled preferences from that user, constructing a
  personalized reward function for them. Additionally, since real-world
  preference data is scarce and challenging to collect at scale, we propose
  careful design choices to construct synthetic preference datasets for
  personalization, generating over 1M synthetic personalized preferences using
  publicly available LLMs. In particular, to successfully transfer from synthetic
  data to real users, we find it crucial for the data to exhibit both high
  diversity and coherent, self-consistent structure. We evaluate FSPO on
  personalized open-ended generation for up to 1,500 synthetic users across
  across three domains: movie reviews, pedagogical adaptation based on
  educational background, and general question answering, along with a controlled
  human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in
  generating responses that are personalized to synthetic users and a 72% winrate
  with real human users in open-ended question answering.
paper_authors._id: 67c01972d63ea6742473aa2a
paper_authors.name: Anikait Singh
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-02-27T07:51:17.284Z
paper_authors._id: 6511ee845b7e52b0251fdee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
paper_authors.fullname: Anikait Singh
paper_authors.type: user
paper_authors.user: Asap7772
paper_authors._id: 67c01972d63ea6742473aa2b
paper_authors.name: Sheryl Hsu
paper_authors._id: 67c01972d63ea6742473aa2c
paper_authors.name: Kyle Hsu
paper_authors._id: 67c01972d63ea6742473aa2d
paper_authors.name: Eric Mitchell
paper_authors._id: 67c01972d63ea6742473aa2e
paper_authors.name: Stefano Ermon
paper_authors._id: 67c01972d63ea6742473aa2f
paper_authors.name: Tatsunori Hashimoto
paper_authors._id: 67c01972d63ea6742473aa30
paper_authors.name: Archit Sharma
paper_authors._id: 67c01972d63ea6742473aa31
paper_authors.name: Chelsea Finn
paper_title: FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in
    LLMs Elicits Effective Personalization to Real Users
paper_summary: Effective personalization of LLMs is critical for a broad range of
  user-interfacing applications such as virtual assistants and content curation.
  Inspired by the strong in-context learning capabilities of LLMs, we propose
  Few-Shot Preference Optimization (FSPO), which reframes reward modeling as a
  meta-learning problem. Under this framework, an LLM learns to quickly adapt to
  a user via a few labeled preferences from that user, constructing a
  personalized reward function for them. Additionally, since real-world
  preference data is scarce and challenging to collect at scale, we propose
  careful design choices to construct synthetic preference datasets for
  personalization, generating over 1M synthetic personalized preferences using
  publicly available LLMs. In particular, to successfully transfer from synthetic
  data to real users, we find it crucial for the data to exhibit both high
  diversity and coherent, self-consistent structure. We evaluate FSPO on
  personalized open-ended generation for up to 1,500 synthetic users across
  across three domains: movie reviews, pedagogical adaptation based on
  educational background, and general question answering, along with a controlled
  human study. Overall, FSPO achieves an 87% Alpaca Eval winrate on average in
  generating responses that are personalized to synthetic users and a 72% winrate
  with real human users in open-ended question answering.
paper_ai_keywords: Few-Shot Preference Optimization (FSPO) in-context learning reward modeling meta-learning user-adaptation preference data synthetic personalized preferences publicly available LLMs high diversity coherent, self-consistent structure open-ended generation Alpaca Eval