title: Reformatted Alignment
summary: The quality of finetuning data is crucial for aligning large language models
  (LLMs) with human values. Current methods to improve data quality are either
  labor-intensive or prone to factual errors caused by LLM hallucinations. This
  paper explores elevating the quality of existing instruction data to better
  align with human values, introducing a simple and effective approach named
  ReAlign, which reformats the responses of instruction data into a format that
  better aligns with pre-established criteria and the collated evidence. This
  approach minimizes human annotation, hallucination, and the difficulty in
  scaling, remaining orthogonal to existing alignment techniques. Experimentally,
  ReAlign significantly boosts the general alignment ability, math reasoning,
  factuality, and readability of the LLMs.
    Encouragingly, without introducing any additional data or advanced training
  techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical
  reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy.
  Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment
  ability measured by the Alpaca dataset. This work highlights the need for
  further research into the science and mechanistic interpretability of LLMs. We
  have made the associated code and data publicly accessible to support future
  studies at https://github.com/GAIR-NLP/ReAlign.
paper_authors._id: 65d43950138129df58b7f3da
paper_authors.name: Run-Ze Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-20T08:38:38.113Z
paper_authors._id: 616bfc2b40e2f69baa1c7add
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616bfc2b40e2f69baa1c7add/Os7_qgMei-2lRVelrOG7B.jpeg
paper_authors.fullname: Run-Ze Fan
paper_authors.type: user
paper_authors.user: Vfrz
paper_authors._id: 65d43950138129df58b7f3db
paper_authors.name: Xuefeng Li
paper_authors._id: 65d43950138129df58b7f3dc
paper_authors.name: Haoyang Zou
paper_authors._id: 65d43950138129df58b7f3dd
paper_authors.name: Junlong Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T07:18:56.997Z
paper_authors._id: 621e40ac944c7e36aaec2369
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/621e40ac944c7e36aaec2369/Yj-FJRWps3rvsS_B2bnKo.jpeg
paper_authors.fullname: Junlong Li
paper_authors.type: user
paper_authors.user: lockon
paper_authors._id: 65d43950138129df58b7f3de
paper_authors.name: Shwai He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:29.509Z
paper_authors._id: 64468d7a3b28db22e7f84418
paper_authors.avatarUrl: /avatars/ff05994acd7da64a2ed6e44f12cf77bc.svg
paper_authors.fullname: Shwai He
paper_authors.type: user
paper_authors.user: Shwai
paper_authors._id: 65d43950138129df58b7f3df
paper_authors.name: Ethan Chern
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-09T09:27:19.140Z
paper_authors._id: 64bb5f9d8e051085bace4d1e
paper_authors.avatarUrl: /avatars/15ccbb78c6131dfe46b7a9d8e7d1a31f.svg
paper_authors.fullname: Ethan Chern
paper_authors.type: user
paper_authors.user: ethanchern
paper_authors._id: 65d43950138129df58b7f3e0
paper_authors.name: Jiewen Hu
paper_authors._id: 65d43950138129df58b7f3e1
paper_authors.name: Pengfei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-02T07:16:31.650Z
paper_authors._id: 6144a0c4ff1146bbd84d9865
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1661715958139-6144a0c4ff1146bbd84d9865.png
paper_authors.fullname: Pengfei Liu
paper_authors.type: user
paper_authors.user: Pengfei
paper_title: Reformatted Alignment
paper_summary: The quality of finetuning data is crucial for aligning large language models
  (LLMs) with human values. Current methods to improve data quality are either
  labor-intensive or prone to factual errors caused by LLM hallucinations. This
  paper explores elevating the quality of existing instruction data to better
  align with human values, introducing a simple and effective approach named
  ReAlign, which reformats the responses of instruction data into a format that
  better aligns with pre-established criteria and the collated evidence. This
  approach minimizes human annotation, hallucination, and the difficulty in
  scaling, remaining orthogonal to existing alignment techniques. Experimentally,
  ReAlign significantly boosts the general alignment ability, math reasoning,
  factuality, and readability of the LLMs.
    Encouragingly, without introducing any additional data or advanced training
  techniques, and merely by reformatting the response, LLaMA-2-13B's mathematical
  reasoning ability on GSM8K can be improved from 46.77% to 56.63% in accuracy.
  Additionally, a mere 5% of ReAlign data yields a 67% boost in general alignment
  ability measured by the Alpaca dataset. This work highlights the need for
  further research into the science and mechanistic interpretability of LLMs. We
  have made the associated code and data publicly accessible to support future
  studies at https://github.com/GAIR-NLP/ReAlign.
paper_ai_keywords: large language models (LLMs) human values instruction data ReAlign pre-established criteria collated evidence hallucination general alignment math reasoning factuality readability LLaMA-2-13B GSM8K Alpaca dataset