Reformatted Alignment
The quality of finetuning data is crucial for aligning large language models
(LLMs) with human values. Current methods to improve data quality are either
labor-intensive or prone to factual errors caused by LLM hallucinations. This
paper explores elevating the quality of existing instruction data to better
align with human values, introducing a simple and effective approach named
ReAlign, which reformats the responses of instruction data into a format that
better aligns with pre-established criteria and the collated evidence. This
approach minimizes human annotation, hallucination, and the difficulty in
scaling, remaining orthogonal to existing alignment techniques. Experimentally,
ReAlign significantly boosts the general alignment ability, math reasoning,
factuality, and readability of the LLMs.
  Encouragingly, without introducing any additional data or advanced training
techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical
reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy.
Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment
ability measured by the Alpaca dataset. This work highlights the need for
further research into the science and mechanistic interpretability of LLMs. We
have made the associated code and data publicly accessible to support future
studies at https://github.com/GAIR-NLP/ReAlign.
65d43950138129df58b7f3da
Run-Ze Fan
claimed_verified
2024-02-20T08:38:38.113Z
616bfc2b40e2f69baa1c7add
https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
Run-Ze Fan
user
Vfrz
65d43950138129df58b7f3db
Xuefeng Li
65d43950138129df58b7f3dc
Haoyang Zou
65d43950138129df58b7f3dd
Junlong Li
claimed_verified
2024-04-23T07:18:56.997Z
621e40ac944c7e36aaec2369
https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
Junlong Li
user
lockon
65d43950138129df58b7f3de
Shwai He
claimed_verified
2024-07-18T09:09:29.509Z
64468d7a3b28db22e7f84418
/avatars/ff05994acd7da64a2ed6e44f12cf77bc.svg
Shwai He
user
Shwai
65d43950138129df58b7f3df
Ethan Chern
claimed_verified
2024-07-09T09:27:19.140Z
64bb5f9d8e051085bace4d1e
/avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
Ethan Chern
user
ethanchern
65d43950138129df58b7f3e0
Jiewen Hu
65d43950138129df58b7f3e1
Pengfei Liu
claimed_verified
2024-05-02T07:16:31.650Z
6144a0c4ff1146bbd84d9865
https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
Pengfei Liu
user
Pengfei
Reformatted Alignment
The quality of finetuning data is crucial for aligning large language models
(LLMs) with human values. Current methods to improve data quality are either
labor-intensive or prone to factual errors caused by LLM hallucinations. This
paper explores elevating the quality of existing instruction data to better
align with human values, introducing a simple and effective approach named
ReAlign, which reformats the responses of instruction data into a format that
better aligns with pre-established criteria and the collated evidence. This
approach minimizes human annotation, hallucination, and the difficulty in
scaling, remaining orthogonal to existing alignment techniques. Experimentally,
ReAlign significantly boosts the general alignment ability, math reasoning,
factuality, and readability of the LLMs.
  Encouragingly, without introducing any additional data or advanced training
techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical
reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy.
Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment
ability measured by the Alpaca dataset. This work highlights the need for
further research into the science and mechanistic interpretability of LLMs. We
have made the associated code and data publicly accessible to support future
studies at https://github.com/GAIR-NLP/ReAlign.
ReAlign, a simple approach that reformats instruction data responses, enhances the quality and alignment of large language models without additional data or advanced techniques.
ReAlign instruction data human values LLM hallucinations general alignment mathematical reasoning factuality readability LLaMA-2-13B GSM8K Alpaca dataset