title: Discovering Preference Optimization Algorithms with and for Large
    Language Models
summary: Offline preference optimization is a key method for enhancing and controlling
  the quality of Large Language Model (LLM) outputs. Typically, preference
  optimization is approached as an offline supervised learning task using
  manually-crafted convex loss functions. While these methods are based on
  theoretical insights, they are inherently constrained by human creativity, so
  the large search space of possible loss functions remains under explored. We
  address this by performing LLM-driven objective discovery to automatically
  discover new state-of-the-art preference optimization algorithms without
  (expert) human intervention. Specifically, we iteratively prompt an LLM to
  propose and implement new preference optimization loss functions based on
  previously-evaluated performance metrics. This process leads to the discovery
  of previously-unknown and performant preference optimization algorithms. The
  best performing of these we call Discovered Preference Optimization (DiscoPOP),
  a novel algorithm that adaptively blends logistic and exponential losses.
  Experiments demonstrate the state-of-the-art performance of DiscoPOP and its
  successful transfer to held-out tasks.
paper_authors._id: 666a470e02ff2c30275f59fe
paper_authors.name: Chris Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-13T09:31:19.745Z
paper_authors._id: 64cbfc1d3576a06fa252736d
paper_authors.avatarUrl: /avatars/7345beafa590d5c9d5ac80cb33a9f0ab.svg
paper_authors.fullname: Chris Lu
paper_authors.type: user
paper_authors.user: chrlu
paper_authors._id: 666a470e02ff2c30275f59ff
paper_authors.name: Samuel Holt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:19:05.864Z
paper_authors._id: 63e69f04b6a40bf941e27e99
paper_authors.avatarUrl: /avatars/79afae1a88068797798dfcbf34db6c45.svg
paper_authors.fullname: Samuel Holt
paper_authors.type: user
paper_authors.user: samuelholt
paper_authors._id: 666a470e02ff2c30275f5a00
paper_authors.name: Claudio Fanconi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:19:11.331Z
paper_authors._id: 63a6f89de27a6dbd485be3a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a6f89de27a6dbd485be3a6/hOPR6bk0CDSWimB7owUmA.jpeg
paper_authors.fullname: Claudio Fanconi
paper_authors.type: user
paper_authors.user: fanconic
paper_authors._id: 666a470e02ff2c30275f5a01
paper_authors.name: Alex J. Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-13T20:19:17.217Z
paper_authors._id: 636c1e4415cd58e915bc45df
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636c1e4415cd58e915bc45df/KnPgdPe0G5ngvXaCBua6R.jpeg
paper_authors.fullname: Alex J. Chan
paper_authors.type: user
paper_authors.user: XanderJC
paper_authors._id: 666a470e02ff2c30275f5a02
paper_authors.name: Jakob Foerster
paper_authors._id: 666a470e02ff2c30275f5a03
paper_authors.name: Mihaela van der Schaar
paper_authors._id: 666a470e02ff2c30275f5a04
paper_authors.name: Robert Tjarko Lange
paper_title: Discovering Preference Optimization Algorithms with and for Large
    Language Models
paper_summary: Offline preference optimization is a key method for enhancing and controlling
  the quality of Large Language Model (LLM) outputs. Typically, preference
  optimization is approached as an offline supervised learning task using
  manually-crafted convex loss functions. While these methods are based on
  theoretical insights, they are inherently constrained by human creativity, so
  the large search space of possible loss functions remains under explored. We
  address this by performing LLM-driven objective discovery to automatically
  discover new state-of-the-art preference optimization algorithms without
  (expert) human intervention. Specifically, we iteratively prompt an LLM to
  propose and implement new preference optimization loss functions based on
  previously-evaluated performance metrics. This process leads to the discovery
  of previously-unknown and performant preference optimization algorithms. The
  best performing of these we call Discovered Preference Optimization (DiscoPOP),
  a novel algorithm that adaptively blends logistic and exponential losses.
  Experiments demonstrate the state-of-the-art performance of DiscoPOP and its
  successful transfer to held-out tasks.
paper_ai_keywords: Large Language Model (LLM) offline supervised learning convex loss functions LLM-driven objective discovery preference optimization loss functions logistic and exponential losses Discovered Preference Optimization (DiscoPOP)