Evaluating the role of `Constitutions' for learning from AI feedback
The growing capabilities of large language models (LLMs) have led to their
use as substitutes for human feedback for training and assessing other LLMs.
These methods often rely on `constitutions', written guidelines which a critic
model uses to provide feedback and improve generations. We investigate how the
choice of constitution affects feedback quality by using four different
constitutions to improve patient-centered communication in medical interviews.
In pairwise comparisons conducted by 215 human raters, we found that detailed
constitutions led to better results regarding emotive qualities. However, none
of the constitutions outperformed the baseline in learning more
practically-oriented skills related to information gathering and provision. Our
findings indicate that while detailed constitutions should be prioritised,
there are possible limitations to the effectiveness of AI feedback as a reward
signal in certain areas.
673c87d3dc27d069a3c808f7
Saskia Redgate
admin_assigned
2024-11-19T16:40:47.383Z
65f1e0e65c4e61a391f087de
/avatars/02248cb36255cd18583ceead0839fd6e.svg
Saskia Redgate
user
saskiar
673c87d3dc27d069a3c808f8
Andrew M. Bean
admin_assigned
2024-11-19T16:40:55.178Z
659bec4728676374f33ef921
/avatars/217ae547d6460e65c6d2a23012741830.svg
Andrew Bean
user
ambean
673c87d3dc27d069a3c808f9
Adam Mahdi
claimed_verified
2024-11-20T10:09:01.008Z
673cf9c93af47d1d2b6584c9
/avatars/0792921a8adaa1493255abb0b5aaa4f1.svg
Adam Mahdi
user
ammaox
Evaluating the role of `Constitutions' for learning from AI feedback
The growing capabilities of large language models (LLMs) have led to their
use as substitutes for human feedback for training and assessing other LLMs.
These methods often rely on `constitutions', written guidelines which a critic
model uses to provide feedback and improve generations. We investigate how the
choice of constitution affects feedback quality by using four different
constitutions to improve patient-centered communication in medical interviews.
In pairwise comparisons conducted by 215 human raters, we found that detailed
constitutions led to better results regarding emotive qualities. However, none
of the constitutions outperformed the baseline in learning more
practically-oriented skills related to information gathering and provision. Our
findings indicate that while detailed constitutions should be prioritised,
there are possible limitations to the effectiveness of AI feedback as a reward
signal in certain areas.
Investigation into the impact of different constitutions on AI-generated feedback shows detailed constitutions improve emotive qualities in medical interviews but not practical skills.
large language models constitutions critic model patient-centered communication medical interviews pairwise comparisons emotive qualities information gathering provision AI feedback reward signal