Axiomatic Preference Modeling for Longform Question Answering
The remarkable abilities of large language models (LLMs) like GPT-4 partially
stem from post-training processes like Reinforcement Learning from Human
Feedback (RLHF) involving human preferences encoded in a reward model. However,
these reward models (RMs) often lack direct knowledge of why, or under what
principles, the preferences annotations were made. In this study, we identify
principles that guide RMs to better align with human preferences, and then
develop an axiomatic framework to generate a rich variety of preference signals
to uphold them. We use these axiomatic signals to train a model for scoring
answers to longform questions. Our approach yields a Preference Model with only
about 220M parameters that agrees with gold human-annotated preference labels
more often than GPT-4. The contributions of this work include: training a
standalone preference model that can score human- and LLM-generated answers on
the same scale; developing an axiomatic framework for generating training data
pairs tailored to certain principles; and showing that a small amount of
axiomatic signals can help small models outperform GPT-4 in preference scoring.
We release our model on huggingface:
https://huggingface.co/corbyrosset/axiomatic_preference_model
656fd25481e5a9a8fcf9768c
Corby Rosset
admin_assigned
2023-12-06T13:19:23.883Z
64ec43147e2ec711a761c594
/avatars/e4fc3de141ff026eee6eaa0b91d34ff1.svg
corby
user
corbyrosset
656fd25481e5a9a8fcf9768d
Guoqing Zheng
admin_assigned
2023-12-06T13:19:29.963Z
6552620be170f85075ed5d24
/avatars/cc1c783b2a6a7efe25dd01420993a1be.svg
Guoqing Zheng
user
zhenggq
656fd25481e5a9a8fcf9768e
Victor Dibia
admin_assigned
2023-12-06T13:19:36.185Z
630439ada362e7e8b51f88e1
https://cdn-avatars.huggingface.co/v1/production/uploads/1661221392331-630439ada362e7e8b51f88e1.jpeg
Victor Dibia
user
victordibia
656fd25481e5a9a8fcf9768f
Ahmed Awadallah
admin_assigned
2023-12-06T13:19:54.580Z
6557a41ee74d1065f7d05c51
/avatars/b29e57fc61d1d3012637f84b82c14818.svg
Ahmed Awadallah
user
AhmedAwadallah
656fd25481e5a9a8fcf97690
Paul Bennett
Axiomatic Preference Modeling for Longform Question Answering
The remarkable abilities of large language models (LLMs) like GPT-4 partially
stem from post-training processes like Reinforcement Learning from Human
Feedback (RLHF) involving human preferences encoded in a reward model. However,
these reward models (RMs) often lack direct knowledge of why, or under what
principles, the preferences annotations were made. In this study, we identify
principles that guide RMs to better align with human preferences, and then
develop an axiomatic framework to generate a rich variety of preference signals
to uphold them. We use these axiomatic signals to train a model for scoring
answers to longform questions. Our approach yields a Preference Model with only
about 220M parameters that agrees with gold human-annotated preference labels
more often than GPT-4. The contributions of this work include: training a
standalone preference model that can score human- and LLM-generated answers on
the same scale; developing an axiomatic framework for generating training data
pairs tailored to certain principles; and showing that a small amount of
axiomatic signals can help small models outperform GPT-4 in preference scoring.
We release our model on huggingface:
https://huggingface.co/corbyrosset/axiomatic_preference_model
A new preference model trained with an axiomatic framework for generating aligned preference signals outperforms GPT-4 in scoring human and LLM-generated answers.
large language models LLMS GPT-4 Reinforcement Learning from Human Feedback RLHF reward model RM axiomatic framework preference scoring human-annotated preference labels