title: LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for
    Enhanced Following of Instructions with Multiple Constraints
summary: Instruction following is a key capability for LLMs. However, recent studies
  have shown that LLMs often struggle with instructions containing multiple
  constraints (e.g. a request to create a social media post "in a funny tone"
  with "no hashtag"). Despite this, most evaluations focus solely on synthetic
  data. To address this, we introduce RealInstruct, the first benchmark designed
  to evaluate LLMs' ability to follow real-world multi-constrained instructions
  by leveraging queries real users asked AI assistants. We also investigate
  model-based evaluation as a cost-effective alternative to human annotation for
  this task. Our findings reveal that even the proprietary GPT-4 model fails to
  meet at least one constraint on over 21% of instructions, highlighting the
  limitations of state-of-the-art models. To address the performance gap between
  open-source and proprietary models, we propose the Decompose, Critique and
  Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to
  follow constraints. DeCRIM works by decomposing the original instruction into a
  list of constraints and using a Critic model to decide when and where the LLM's
  response needs refinement. Our results show that DeCRIM improves Mistral's
  performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  Moreover, we demonstrate that with strong feedback, open-source LLMs with
  DeCRIM can outperform GPT-4 on both benchmarks.
paper_authors._id: 6707dec1fd8b5c5c50646e45
paper_authors.name: Thomas Palmeira Ferraz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T14:15:37.467Z
paper_authors._id: 60db4e209f0b7223ea4560af
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1624985089254-noauth.jpeg
paper_authors.fullname: Thomas Ferraz
paper_authors.type: user
paper_authors.user: thomas-ferraz
paper_authors._id: 6707dec1fd8b5c5c50646e46
paper_authors.name: Kartik Mehta
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:43:59.085Z
paper_authors._id: 610e5127af1badecc0a80e98
paper_authors.avatarUrl: /avatars/51a631bec84691de0de6b1ca1bc00f1b.svg
paper_authors.fullname: Kartik Mehta
paper_authors.type: user
paper_authors.user: kartikmehtaiitd
paper_authors._id: 6707dec1fd8b5c5c50646e47
paper_authors.name: Yu-Hsiang Lin
paper_authors._id: 6707dec1fd8b5c5c50646e48
paper_authors.name: Haw-Shiuan Chang
paper_authors._id: 6707dec1fd8b5c5c50646e49
paper_authors.name: Shereen Oraby
paper_authors._id: 6707dec1fd8b5c5c50646e4a
paper_authors.name: Sijia Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:46.203Z
paper_authors._id: 651d2b485d3519c0b7595af7
paper_authors.avatarUrl: /avatars/00ce2ecbc35e22a90f72b9015299aa29.svg
paper_authors.fullname: Sijia L
paper_authors.type: user
paper_authors.user: sijial430
paper_authors._id: 6707dec1fd8b5c5c50646e4b
paper_authors.name: Vivek Subramanian
paper_authors._id: 6707dec1fd8b5c5c50646e4c
paper_authors.name: Tagyoung Chung
paper_authors._id: 6707dec1fd8b5c5c50646e4d
paper_authors.name: Mohit Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-10T14:16:23.705Z
paper_authors._id: 665d9d3a057f7c508f98c625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/665d9d3a057f7c508f98c625/u1R9P9sJoAl4zEIcetbPy.jpeg
paper_authors.fullname: Mohit Bansal
paper_authors.type: user
paper_authors.user: mohitbansal
paper_authors._id: 6707dec1fd8b5c5c50646e4e
paper_authors.name: Nanyun Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:44:01.015Z
paper_authors._id: 64c13de54cbd12e168f09a58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c13de54cbd12e168f09a58/Aw5BaNzf6uuwqQ333OQ5y.png
paper_authors.fullname: NANYUN (Violet) PENG
paper_authors.type: user
paper_authors.user: violetpeng
paper_title: LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for
    Enhanced Following of Instructions with Multiple Constraints
paper_summary: Instruction following is a key capability for LLMs. However, recent studies
  have shown that LLMs often struggle with instructions containing multiple
  constraints (e.g. a request to create a social media post "in a funny tone"
  with "no hashtag"). Despite this, most evaluations focus solely on synthetic
  data. To address this, we introduce RealInstruct, the first benchmark designed
  to evaluate LLMs' ability to follow real-world multi-constrained instructions
  by leveraging queries real users asked AI assistants. We also investigate
  model-based evaluation as a cost-effective alternative to human annotation for
  this task. Our findings reveal that even the proprietary GPT-4 model fails to
  meet at least one constraint on over 21% of instructions, highlighting the
  limitations of state-of-the-art models. To address the performance gap between
  open-source and proprietary models, we propose the Decompose, Critique and
  Refine (DeCRIM) self-correction pipeline, which enhances LLMs' ability to
  follow constraints. DeCRIM works by decomposing the original instruction into a
  list of constraints and using a Critic model to decide when and where the LLM's
  response needs refinement. Our results show that DeCRIM improves Mistral's
  performance by 7.3% on RealInstruct and 8.0% on IFEval even with weak feedback.
  Moreover, we demonstrate that with strong feedback, open-source LLMs with
  DeCRIM can outperform GPT-4 on both benchmarks.
paper_ai_keywords: RealInstruct benchmark RealInstruct model-based evaluation human annotation Decompose, Critique and Refine (DeCRIM) self-correction pipeline Critic model Mistral