title: Hybrid Preferences: Learning to Route Instances for Human vs. AI
    Feedback
summary: Learning from human feedback has enabled the alignment of language models
  (LMs) with human preferences. However, directly collecting human preferences
  can be expensive, time-consuming, and can have high variance. An appealing
  alternative is to distill preferences from LMs as a source of synthetic
  annotations as they are more consistent, cheaper, and scale better than human
  annotation; however, they are also prone to biases and errors. In this work, we
  introduce a routing framework that combines inputs from humans and LMs to
  achieve better annotation quality, while reducing the total cost of human
  annotation. The crux of our approach is to identify preference instances that
  will benefit from human annotations. We formulate this as an optimization
  problem: given a preference dataset and an evaluation metric, we train a
  performance prediction model to predict a reward model's performance on an
  arbitrary combination of human and LM annotations and employ a routing strategy
  that selects a combination that maximizes predicted performance. We train the
  performance prediction model on MultiPref, a new preference dataset with 10K
  instances paired with human and LM labels. We show that the selected hybrid
  mixture of LM and direct human preferences using our routing framework achieves
  better reward model performance compared to using either one exclusively. We
  simulate selective human preference collection on three other datasets and show
  that our method generalizes well to all three. We analyze features from the
  routing model to identify characteristics of instances that can benefit from
  human feedback, e.g., prompts with a moderate safety concern or moderate intent
  complexity. We release the dataset, annotation platform, and source code used
  in this study to foster more efficient and accurate preference collection in
  the future.
paper_authors._id: 671ed82fb91f5a4a244a9d87
paper_authors.name: Lester James V. Miranda
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T10:33:55.165Z
paper_authors._id: 634e20a0c1ce28f1de920cc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666064515342-noauth.jpeg
paper_authors.fullname: Lj V. Miranda
paper_authors.type: user
paper_authors.user: ljvmiranda921
paper_authors._id: 671ed82fb91f5a4a244a9d88
paper_authors.name: Yizhong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:40:01.833Z
paper_authors._id: 6269be67d1ac0cde592aba29
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6269be67d1ac0cde592aba29/EP3zZbD7-jWK9ITvHGbnZ.jpeg
paper_authors.fullname: Yizhong Wang
paper_authors.type: user
paper_authors.user: yizhongw
paper_authors._id: 671ed82fb91f5a4a244a9d89
paper_authors.name: Yanai Elazar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:40:23.763Z
paper_authors._id: 623ca115a795593324c4353f
paper_authors.avatarUrl: /avatars/bf11fe728df2786d52ed4d2de12b48d3.svg
paper_authors.fullname: Yanai Elazar
paper_authors.type: user
paper_authors.user: yanaiela
paper_authors._id: 671ed82fb91f5a4a244a9d8a
paper_authors.name: Sachin Kumar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:41:48.584Z
paper_authors._id: 63f24d2d7ddf724fbcc0ea9c
paper_authors.avatarUrl: /avatars/3e24c1aa9c1b4066d2dd56aeb4b0f62e.svg
paper_authors.fullname: sachin kumar
paper_authors.type: user
paper_authors.user: sachinkumar
paper_authors._id: 671ed82fb91f5a4a244a9d8b
paper_authors.name: Valentina Pyatkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:41:55.289Z
paper_authors._id: 6556cff80e7a7067a934445f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6556cff80e7a7067a934445f/PoT7qQ6tVqLGGrYvGBbr8.jpeg
paper_authors.fullname: Valentina Pyatkin
paper_authors.type: user
paper_authors.user: valpy
paper_authors._id: 671ed82fb91f5a4a244a9d8c
paper_authors.name: Faeze Brahman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:42:00.686Z
paper_authors._id: 65282b8d578679aac7888aec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65282b8d578679aac7888aec/dibBkhH-z1c70mJZZxJ7u.jpeg
paper_authors.fullname: Faeze Brahman
paper_authors.type: user
paper_authors.user: faezeb
paper_authors._id: 671ed82fb91f5a4a244a9d8d
paper_authors.name: Noah A. Smith
paper_authors._id: 671ed82fb91f5a4a244a9d8e
paper_authors.name: Hannaneh Hajishirzi
paper_authors._id: 671ed82fb91f5a4a244a9d8f
paper_authors.name: Pradeep Dasigi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:42:06.735Z
paper_authors._id: 6408fcc93461c51cf735a61e
paper_authors.avatarUrl: /avatars/619f3653911d111f046a5a6c30fc8319.svg
paper_authors.fullname: Pradeep Dasigi
paper_authors.type: user
paper_authors.user: pradeepd
paper_title: Hybrid Preferences: Learning to Route Instances for Human vs. AI
    Feedback
paper_summary: Learning from human feedback has enabled the alignment of language models
  (LMs) with human preferences. However, directly collecting human preferences
  can be expensive, time-consuming, and can have high variance. An appealing
  alternative is to distill preferences from LMs as a source of synthetic
  annotations as they are more consistent, cheaper, and scale better than human
  annotation; however, they are also prone to biases and errors. In this work, we
  introduce a routing framework that combines inputs from humans and LMs to
  achieve better annotation quality, while reducing the total cost of human
  annotation. The crux of our approach is to identify preference instances that
  will benefit from human annotations. We formulate this as an optimization
  problem: given a preference dataset and an evaluation metric, we train a
  performance prediction model to predict a reward model's performance on an
  arbitrary combination of human and LM annotations and employ a routing strategy
  that selects a combination that maximizes predicted performance. We train the
  performance prediction model on MultiPref, a new preference dataset with 10K
  instances paired with human and LM labels. We show that the selected hybrid
  mixture of LM and direct human preferences using our routing framework achieves
  better reward model performance compared to using either one exclusively. We
  simulate selective human preference collection on three other datasets and show
  that our method generalizes well to all three. We analyze features from the
  routing model to identify characteristics of instances that can benefit from
  human feedback, e.g., prompts with a moderate safety concern or moderate intent
  complexity. We release the dataset, annotation platform, and source code used
  in this study to foster more efficient and accurate preference collection in
  the future.
paper_ai_keywords: Learning from human feedback Language models (LMs) Synthetic annotations Latent diffusion models (LDMs) Routing framework Preference instances Performance prediction model Reward model Optimization problem Prompt safety concerns Intent complexity Preference dataset Annotation platform Source code