title: Axiomatic Preference Modeling for Longform Question Answering
summary: The remarkable abilities of large language models (LLMs) like GPT-4 partially
  stem from post-training processes like Reinforcement Learning from Human
  Feedback (RLHF) involving human preferences encoded in a reward model. However,
  these reward models (RMs) often lack direct knowledge of why, or under what
  principles, the preferences annotations were made. In this study, we identify
  principles that guide RMs to better align with human preferences, and then
  develop an axiomatic framework to generate a rich variety of preference signals
  to uphold them. We use these axiomatic signals to train a model for scoring
  answers to longform questions. Our approach yields a Preference Model with only
  about 220M parameters that agrees with gold human-annotated preference labels
  more often than GPT-4. The contributions of this work include: training a
  standalone preference model that can score human- and LLM-generated answers on
  the same scale; developing an axiomatic framework for generating training data
  pairs tailored to certain principles; and showing that a small amount of
  axiomatic signals can help small models outperform GPT-4 in preference scoring.
  We release our model on huggingface:
  https://huggingface.co/corbyrosset/axiomatic_preference_model
paper_authors._id: 656fd25481e5a9a8fcf9768c
paper_authors.name: Corby Rosset
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:19:23.883Z
paper_authors._id: 64ec43147e2ec711a761c594
paper_authors.avatarUrl: /avatars/e4fc3de141ff026eee6eaa0b91d34ff1.svg
paper_authors.fullname: corby
paper_authors.type: user
paper_authors.user: corbyrosset
paper_authors._id: 656fd25481e5a9a8fcf9768d
paper_authors.name: Guoqing Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:19:29.963Z
paper_authors._id: 6552620be170f85075ed5d24
paper_authors.avatarUrl: /avatars/cc1c783b2a6a7efe25dd01420993a1be.svg
paper_authors.fullname: Guoqing Zheng
paper_authors.type: user
paper_authors.user: zhenggq
paper_authors._id: 656fd25481e5a9a8fcf9768e
paper_authors.name: Victor Dibia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:19:36.185Z
paper_authors._id: 630439ada362e7e8b51f88e1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661221392331-630439ada362e7e8b51f88e1.jpeg
paper_authors.fullname: Victor Dibia
paper_authors.type: user
paper_authors.user: victordibia
paper_authors._id: 656fd25481e5a9a8fcf9768f
paper_authors.name: Ahmed Awadallah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-06T13:19:54.580Z
paper_authors._id: 6557a41ee74d1065f7d05c51
paper_authors.avatarUrl: /avatars/b29e57fc61d1d3012637f84b82c14818.svg
paper_authors.fullname: Ahmed Awadallah
paper_authors.type: user
paper_authors.user: AhmedAwadallah
paper_authors._id: 656fd25481e5a9a8fcf97690
paper_authors.name: Paul Bennett
paper_title: Axiomatic Preference Modeling for Longform Question Answering
paper_summary: The remarkable abilities of large language models (LLMs) like GPT-4 partially
  stem from post-training processes like Reinforcement Learning from Human
  Feedback (RLHF) involving human preferences encoded in a reward model. However,
  these reward models (RMs) often lack direct knowledge of why, or under what
  principles, the preferences annotations were made. In this study, we identify
  principles that guide RMs to better align with human preferences, and then
  develop an axiomatic framework to generate a rich variety of preference signals
  to uphold them. We use these axiomatic signals to train a model for scoring
  answers to longform questions. Our approach yields a Preference Model with only
  about 220M parameters that agrees with gold human-annotated preference labels
  more often than GPT-4. The contributions of this work include: training a
  standalone preference model that can score human- and LLM-generated answers on
  the same scale; developing an axiomatic framework for generating training data
  pairs tailored to certain principles; and showing that a small amount of
  axiomatic signals can help small models outperform GPT-4 in preference scoring.
  We release our model on huggingface:
  https://huggingface.co/corbyrosset/axiomatic_preference_model
paper_ai_keywords: large language models (LLMs) GPT-4 Reinforcement Learning from Human Feedback (RLHF) reward model (RM) preference signals Preference Model axiomatic framework scoring answers