title: OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of
    Instruction Data
summary: Instruction fine-tuning pretrained LLMs for diverse downstream tasks has
  demonstrated remarkable success and has captured the interest of both academics
  and practitioners. To ensure such fine-tuned LLMs align with human preferences,
  techniques such as RLHF and DPO have emerged. At the same time, there is
  increasing interest in smaller parameter counts for models. In this work, using
  OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the
  OpenBezoar family of models. In this recipe: We first generate synthetic
  instruction fine-tuning data using an open and commercially non-restrictive
  instruction fine-tuned variant of the Falcon-40B model under three schemes
  based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a
  seed dataset) and Orca (with the Flan Collection as a seed dataset), then
  filter these generations using GPT-4 as a human proxy. We then perform
  cost-effective QLoRA-based supervised fine-tuning sequentially with each
  scheme. The resulting checkpoint is further fine-tuned with a subset of the
  HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to
  obtain the final checkpoint. Evaluation is done with the LM Eval Harness
  tasks/metrics as well as on MT-Bench using the "LLM-as-a-judge" framework with
  Claude 2.1, with the finding that the final checkpoint,
  "OpenBezoar-HH-RLHF-DPO", demonstrates superior performance over many models at
  the 3B parameter scale, even outperforming the top model in one of the
  categories on the Huggingface Open LLM Leaderboard. We release
  "OpenBezoar-SFT", "OpenBezoar-HH-RLHF-SFT", "OpenBezoar-HH-RLHF-DPO"
  checkpoints, alongside our generated datasets on HuggingFace at
  https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc
  and our codebase at
  https://bitbucket.org/paladinanalytics/workspace/projects/OP.
paper_authors._id: 6621c753d057cd6d668bad28
paper_authors.name: Chandeepa Dissanayake
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:38:48.032Z
paper_authors._id: 64b65956de30803f4e9a6e46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b65956de30803f4e9a6e46/044T6a04Ekx0fFStOaKiB.jpeg
paper_authors.fullname: Chandeepa Dissanayake
paper_authors.type: user
paper_authors.user: chansurgeplus
paper_authors._id: 6621c753d057cd6d668bad29
paper_authors.name: Lahiru Lowe
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-24T18:23:45.338Z
paper_authors._id: 66293de6fa14eaccf04382cf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66293de6fa14eaccf04382cf/SjNe5po9O2hWKzoL94UGc.png
paper_authors.fullname: Lahiru Lowe
paper_authors.type: user
paper_authors.user: LahiruTLowe
paper_authors._id: 6621c753d057cd6d668bad2a
paper_authors.name: Sachith Gunasekara
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T07:38:45.078Z
paper_authors._id: 647eff9aaa8c04bbf9365219
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647eff9aaa8c04bbf9365219/lu8O13Fgsferal-yhAtih.jpeg
paper_authors.fullname: Sachith Gunasekara
paper_authors.type: user
paper_authors.user: sachithgunasekara
paper_authors._id: 6621c753d057cd6d668bad2b
paper_authors.name: Yasiru Ratnayake
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-19T11:17:25.664Z
paper_authors._id: 6440fa0b66a62c605c91d739
paper_authors.avatarUrl: /avatars/3b941c8d2a7941bb77f46167907df704.svg
paper_authors.fullname: Yasiru Dhanika Ratnayake
paper_authors.type: user
paper_authors.user: yasiru89
paper_title: OpenBezoar: Small, Cost-Effective and Open Models Trained on Mixes of
    Instruction Data
paper_summary: Instruction fine-tuning pretrained LLMs for diverse downstream tasks has
  demonstrated remarkable success and has captured the interest of both academics
  and practitioners. To ensure such fine-tuned LLMs align with human preferences,
  techniques such as RLHF and DPO have emerged. At the same time, there is
  increasing interest in smaller parameter counts for models. In this work, using
  OpenLLaMA 3Bv2 as a base model, we describe the recipe used to fine-tune the
  OpenBezoar family of models. In this recipe: We first generate synthetic
  instruction fine-tuning data using an open and commercially non-restrictive
  instruction fine-tuned variant of the Falcon-40B model under three schemes
  based on: LaMini-LM, WizardLM/Evol-Instruct (with databricks-dolly-15k as a
  seed dataset) and Orca (with the Flan Collection as a seed dataset), then
  filter these generations using GPT-4 as a human proxy. We then perform
  cost-effective QLoRA-based supervised fine-tuning sequentially with each
  scheme. The resulting checkpoint is further fine-tuned with a subset of the
  HH-RLHF dataset to minimize distribution shift prior to using the DPO loss to
  obtain the final checkpoint. Evaluation is done with the LM Eval Harness
  tasks/metrics as well as on MT-Bench using the "LLM-as-a-judge" framework with
  Claude 2.1, with the finding that the final checkpoint,
  "OpenBezoar-HH-RLHF-DPO", demonstrates superior performance over many models at
  the 3B parameter scale, even outperforming the top model in one of the
  categories on the Huggingface Open LLM Leaderboard. We release
  "OpenBezoar-SFT", "OpenBezoar-HH-RLHF-SFT", "OpenBezoar-HH-RLHF-DPO"
  checkpoints, alongside our generated datasets on HuggingFace at
  https://huggingface.co/collections/SurgeGlobal/open-bezoar-6620a24923e12127e9e2b9cc
  and our codebase at
  https://bitbucket.org/paladinanalytics/workspace/projects/OP.
paper_ai_keywords: RLHF DPO OpenLLaMA 3Bv2 OpenBezoar .synthetic instruction fine-tuning data LaMini-LM WizardLM/Evol-Instruct databricks-dolly-15k Orca Flan Collection GPT-4 QLoRA supervised fine-tuning HH-RLHF dataset LM Eval Harness MT-Bench "LLM-as-a-judge" framework Claude 2.1 OpenBezoar-HH-RLHF-DPO OpenBezoar-SFT OpenBezoar-HH-RLHF-SFT