title: Teaching Language Models to Self-Improve through Interactive
    Demonstrations
summary: The self-improving ability of large language models (LLMs), enabled by
  prompting them to analyze and revise their own outputs, has garnered
  significant interest in recent research. However, this ability has been shown
  to be absent and difficult to learn for smaller models, thus widening the
  performance gap between state-of-the-art LLMs and more cost-effective and
  faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
  that endows smaller models with such self-improvement ability, and show that
  our approach can improve a LLaMA-7b's performance on math and reasoning tasks
  by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
  model to interact with LLMs to collect feedback and improvements on its own
  generations. We then replay this experience to train the small model. Our
  experiments on four math and reasoning datasets show that the interactive
  experience of learning from and correcting its own mistakes is crucial for
  small models to improve their performance.
paper_authors._id: 6535eeb8e983fb23fa695eda
paper_authors.name: Xiao Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-23T14:46:35.296Z
paper_authors._id: 6234fd736dcfc5fe9f5b8601
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647639915850-noauth.jpeg
paper_authors.fullname: Xiao Yu
paper_authors.type: user
paper_authors.user: jasonyux
paper_authors._id: 6535eeb8e983fb23fa695edb
paper_authors.name: Baolin Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T07:40:35.014Z
paper_authors._id: 61942296d5c2ba6daa290357
paper_authors.avatarUrl: /avatars/594021cc183c4922d48b46f43772a062.svg
paper_authors.fullname: Baolin Peng
paper_authors.type: user
paper_authors.user: Baolin
paper_authors._id: 6535eeb8e983fb23fa695edc
paper_authors.name: Michel Galley
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T07:40:47.113Z
paper_authors._id: 5f6bed855e78cc6b0ed31dee
paper_authors.avatarUrl: /avatars/d67edb079fa3fe7ea6f2081f7a3afe9e.svg
paper_authors.fullname: Michel Galley
paper_authors.type: user
paper_authors.user: mgalley
paper_authors._id: 6535eeb8e983fb23fa695edd
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-23T07:40:54.789Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 6535eeb8e983fb23fa695ede
paper_authors.name: Zhou Yu
paper_title: Teaching Language Models to Self-Improve through Interactive
    Demonstrations
paper_summary: The self-improving ability of large language models (LLMs), enabled by
  prompting them to analyze and revise their own outputs, has garnered
  significant interest in recent research. However, this ability has been shown
  to be absent and difficult to learn for smaller models, thus widening the
  performance gap between state-of-the-art LLMs and more cost-effective and
  faster ones. To reduce this gap, we introduce TriPosT, a training algorithm
  that endows smaller models with such self-improvement ability, and show that
  our approach can improve a LLaMA-7b's performance on math and reasoning tasks
  by up to 7.13%. In contrast to prior work, we achieve this by using the smaller
  model to interact with LLMs to collect feedback and improvements on its own
  generations. We then replay this experience to train the small model. Our
  experiments on four math and reasoning datasets show that the interactive
  experience of learning from and correcting its own mistakes is crucial for
  small models to improve their performance.
paper_ai_summary: TriPosT trains smaller models to self-improve by interacting with larger language models to collect feedback, narrowing the performance gap in math and reasoning tasks.
paper_ai_keywords: large language models self-improving ability prompting smaller models TriPosT training algorithm LLaMA-7b math tasks reasoning tasks interactive experience datasets