title: ICE-GRT: Instruction Context Enhancement by Generative Reinforcement
    based Transformers
summary: The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA
  encounter limitations in domain-specific tasks, with these models often lacking
  depth and accuracy in specialized areas, and exhibiting a decrease in general
  capabilities when fine-tuned, particularly analysis ability in small sized
  models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement
  Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization
  (PPO), demonstrating remarkable ability in in-domain scenarios without
  compromising general task performance. Our exploration of ICE-GRT highlights
  its understanding and reasoning ability to not only generate robust answers but
  also to provide detailed analyses of the reasons behind the answer. This
  capability marks a significant progression beyond the scope of Supervised
  Fine-Tuning models. The success of ICE-GRT is dependent on several crucial
  factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage
  Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in
  domain-specific tasks and across 12 general Language tasks against equivalent
  size and even larger size LLMs, highlighting the effectiveness of our approach.
  We provide a comprehensive analysis of the ICE-GRT, underscoring the
  significant advancements it brings to the field of LLM.
paper_authors._id: 65976a9e2d23530ec073898c
paper_authors.name: Chen Zheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:21:06.000Z
paper_authors._id: 63be66194a2beec6555d49d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63be66194a2beec6555d49d6/Ck-VAuADj9BCDxU0NsdTF.jpeg
paper_authors.fullname: CHEN ZHENG
paper_authors.type: user
paper_authors.user: zhengchenphd
paper_authors._id: 65976a9e2d23530ec073898d
paper_authors.name: Ke Sun
paper_authors._id: 65976a9e2d23530ec073898e
paper_authors.name: Da Tang
paper_authors._id: 65976a9e2d23530ec073898f
paper_authors.name: Yukun Ma
paper_authors._id: 65976a9e2d23530ec0738990
paper_authors.name: Yuyu Zhang
paper_authors._id: 65976a9e2d23530ec0738991
paper_authors.name: Chenguang Xi
paper_authors._id: 65976a9e2d23530ec0738992
paper_authors.name: Xun Zhou
paper_title: ICE-GRT: Instruction Context Enhancement by Generative Reinforcement
    based Transformers
paper_summary: The emergence of Large Language Models (LLMs) such as ChatGPT and LLaMA
  encounter limitations in domain-specific tasks, with these models often lacking
  depth and accuracy in specialized areas, and exhibiting a decrease in general
  capabilities when fine-tuned, particularly analysis ability in small sized
  models. To address these gaps, we introduce ICE-GRT, utilizing Reinforcement
  Learning from Human Feedback (RLHF) grounded in Proximal Policy Optimization
  (PPO), demonstrating remarkable ability in in-domain scenarios without
  compromising general task performance. Our exploration of ICE-GRT highlights
  its understanding and reasoning ability to not only generate robust answers but
  also to provide detailed analyses of the reasons behind the answer. This
  capability marks a significant progression beyond the scope of Supervised
  Fine-Tuning models. The success of ICE-GRT is dependent on several crucial
  factors, including Appropriate Data, Reward Size Scaling, KL-Control, Advantage
  Normalization, etc. The ICE-GRT model exhibits state-of-the-art performance in
  domain-specific tasks and across 12 general Language tasks against equivalent
  size and even larger size LLMs, highlighting the effectiveness of our approach.
  We provide a comprehensive analysis of the ICE-GRT, underscoring the
  significant advancements it brings to the field of LLM.
paper_ai_keywords: Reinforcement Learning from Human Feedback (RLHF) Proximal Policy Optimization (PPO) in-domain scenarios Supervised Fine-Tuning KL-Control Advantage Normalization state-of-the-art performance