title: REFINE-AF: A Task-Agnostic Framework to Align Language Models via
    Self-Generated Instructions using Reinforcement Learning from Automated
    Feedback
summary: Instruction-based Large Language Models (LLMs) have proven effective in
  numerous few-shot or zero-shot Natural Language Processing (NLP) tasks.
  However, creating human-annotated instruction data is time-consuming,
  expensive, and often limited in quantity and task diversity. Previous research
  endeavors have attempted to address this challenge by proposing frameworks
  capable of generating instructions in a semi-automated and task-agnostic manner
  directly from the model itself. Many of these efforts have relied on large
  API-only parameter-based models such as GPT-3.5 (175B), which are expensive,
  and subject to limits on a number of queries. This paper explores the
  performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B,
  and Mistral 7B, using a semi-automated framework, thereby reducing human
  intervention, effort, and cost required to generate an instruction dataset for
  fine-tuning LLMs. Furthermore, we demonstrate that incorporating a
  Reinforcement Learning (RL) based training algorithm into this LLMs-based
  framework leads to further enhancements. Our evaluation of the dataset reveals
  that these RL-based frameworks achieve a substantial improvements in 63-66% of
  the tasks compared to previous approaches.
paper_authors._id: 6822a0f7f90521b5d8cb4573
paper_authors.name: Aniruddha Roy
paper_authors._id: 6822a0f7f90521b5d8cb4574
paper_authors.name: Pretam Ray
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:32:17.365Z
paper_authors._id: 62d3ac975a869f4338852b6f
paper_authors.avatarUrl: /avatars/4094eb1ea1244b0a9b60344528efa25a.svg
paper_authors.fullname: Pretam Ray
paper_authors.type: user
paper_authors.user: Pretam
paper_authors._id: 6822a0f7f90521b5d8cb4575
paper_authors.name: Abhilash Nandy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-13T08:32:23.396Z
paper_authors._id: 5f89da6c5d083370c711f37c
paper_authors.avatarUrl: /avatars/c2f17a4a636973817fd5da2ae6dbaac3.svg
paper_authors.fullname: Abhilash Nandy
paper_authors.type: user
paper_authors.user: abhi1nandy2
paper_authors._id: 6822a0f7f90521b5d8cb4576
paper_authors.name: Somak Aditya
paper_authors._id: 6822a0f7f90521b5d8cb4577
paper_authors.name: Pawan Goyal
paper_title: REFINE-AF: A Task-Agnostic Framework to Align Language Models via
    Self-Generated Instructions using Reinforcement Learning from Automated
    Feedback
paper_summary: Instruction-based Large Language Models (LLMs) have proven effective in
  numerous few-shot or zero-shot Natural Language Processing (NLP) tasks.
  However, creating human-annotated instruction data is time-consuming,
  expensive, and often limited in quantity and task diversity. Previous research
  endeavors have attempted to address this challenge by proposing frameworks
  capable of generating instructions in a semi-automated and task-agnostic manner
  directly from the model itself. Many of these efforts have relied on large
  API-only parameter-based models such as GPT-3.5 (175B), which are expensive,
  and subject to limits on a number of queries. This paper explores the
  performance of three open-source small LLMs such as LLaMA 2-7B, LLama 2-13B,
  and Mistral 7B, using a semi-automated framework, thereby reducing human
  intervention, effort, and cost required to generate an instruction dataset for
  fine-tuning LLMs. Furthermore, we demonstrate that incorporating a
  Reinforcement Learning (RL) based training algorithm into this LLMs-based
  framework leads to further enhancements. Our evaluation of the dataset reveals
  that these RL-based frameworks achieve a substantial improvements in 63-66% of
  the tasks compared to previous approaches.
paper_ai_summary: Semi-automated frameworks using open-source small LLMs and reinforcement learning significantly improve instruction dataset generation for LLM fine-tuning across various tasks.
paper_ai_keywords: Instruction-based Large Language Models (LLMs) Natural Language Processing (NLP) few-shot zero-shot semi-automated framework task-agnostic GPT-3.5 LLaMA 2-7B LLama 2-13B Mistral 7B Reinforcement Learning (RL) fine-tuning