title: Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning
    Enhancement in RLHF and Effective-Merged LLMs
summary: Despite the advances in Large Language Models (LLMs), exemplified by models
  like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often
  struggle with generating in-depth and coherent dialogues. This paper presents a
  novel two-step Coarse-to-Fine Actor model to address the inherent limitations
  in conversational and analytical capabilities of small-sized LLMs. Our approach
  begins with the Policy-based Coarse Actor, employing a technique we term
  "Continuous Maximization". The Coarse Actor establishes an enhanced,
  knowledge-rich pool adept at aligning with human preference styles in analysis
  and reasoning. Through the RLHF process, it employs Continuous Maximization, a
  strategy that dynamically and adaptively extends the output length limit,
  enabling the generation of more detailed and analytical content. Subsequently,
  the Fine Actor refines this analytical content, addressing the generation of
  excessively redundant information from the Coarse Actor. We introduce a
  "Knowledge Residue Merger" approach, refining the content from the Coarse Actor
  and merging it with an existing Instruction model to improve quality,
  correctness, and reduce redundancies. We applied our methodology to the popular
  Mistral model, creating Mistral-C2F, which has demonstrated exceptional
  performance across 11 general language tasks and the MT-Bench Dialogue task,
  outperforming similar-scale models and even larger models with 13B and 30B
  parameters. Our model has significantly improved conversational and analytical
  reasoning abilities.
paper_authors._id: 666b9e39c636d47bd57e7ff4
paper_authors.name: Chen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:20:15.429Z
paper_authors._id: 63be66194a2beec6555d49d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63be66194a2beec6555d49d6/Ck-VAuADj9BCDxU0NsdTF.jpeg
paper_authors.fullname: CHEN ZHENG
paper_authors.type: user
paper_authors.user: zhengchenphd
paper_authors._id: 666b9e39c636d47bd57e7ff5
paper_authors.name: Ke Sun
paper_authors._id: 666b9e39c636d47bd57e7ff6
paper_authors.name: Xun Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T11:40:08.786Z
paper_authors._id: 62533db4a06ec75172eeabe7
paper_authors.avatarUrl: /avatars/b1a4dad90afae5c00df97233a97777db.svg
paper_authors.fullname: xunzhou
paper_authors.type: user
paper_authors.user: xunzhou
paper_title: Mistral-C2F: Coarse to Fine Actor for Analytical and Reasoning
    Enhancement in RLHF and Effective-Merged LLMs
paper_summary: Despite the advances in Large Language Models (LLMs), exemplified by models
  like GPT-4 and Claude, smaller-scale LLMs such as Llama and Mistral often
  struggle with generating in-depth and coherent dialogues. This paper presents a
  novel two-step Coarse-to-Fine Actor model to address the inherent limitations
  in conversational and analytical capabilities of small-sized LLMs. Our approach
  begins with the Policy-based Coarse Actor, employing a technique we term
  "Continuous Maximization". The Coarse Actor establishes an enhanced,
  knowledge-rich pool adept at aligning with human preference styles in analysis
  and reasoning. Through the RLHF process, it employs Continuous Maximization, a
  strategy that dynamically and adaptively extends the output length limit,
  enabling the generation of more detailed and analytical content. Subsequently,
  the Fine Actor refines this analytical content, addressing the generation of
  excessively redundant information from the Coarse Actor. We introduce a
  "Knowledge Residue Merger" approach, refining the content from the Coarse Actor
  and merging it with an existing Instruction model to improve quality,
  correctness, and reduce redundancies. We applied our methodology to the popular
  Mistral model, creating Mistral-C2F, which has demonstrated exceptional
  performance across 11 general language tasks and the MT-Bench Dialogue task,
  outperforming similar-scale models and even larger models with 13B and 30B
  parameters. Our model has significantly improved conversational and analytical
  reasoning abilities.
paper_ai_summary: A two-step Coarse-to-Fine Actor model enhances small-sized LLMs' conversational and analytical capabilities through Continuous Maximization and Knowledge Residue Merger, achieving superior performance in various tasks.
paper_ai_keywords: Large Language Models LLMs GPT-4 Claude Llama Mistral Coarse-to-Fine Actor model Policy-based Coarse Actor Continuous Maximization RLHF Fine Actor Knowledge Residue Merger Mistral-C2F MT-Bench Dialogue task