title: Platypus: Quick, Cheap, and Powerful Refinement of LLMs
summary: We present Platypus, a family of fine-tuned and merged Large
  Language Models (LLMs) that achieves the strongest performance and currently
  stands at first place in HuggingFace's Open LLM Leaderboard as of the release
  date of this work. In this work we describe (1) our curated dataset
  Open-Platypus, that is a subset of other open datasets and which
  we release to the public (2) our process of fine-tuning and merging
  LoRA modules in order to conserve the strong prior of pretrained LLMs, while
  bringing specific domain knowledge to the surface (3) our efforts in checking
  for test data leaks and contamination in the training data, which can inform
  future research. Specifically, the Platypus family achieves strong performance
  in quantitative LLM metrics across model sizes, topping the global Open LLM
  leaderboard while using just a fraction of the fine-tuning data and overall
  compute that are required for other state-of-the-art fine-tuned LLMs. In
  particular, a 13B Platypus model can be trained on a single A100 GPU
  using 25k questions in 5 hours. This is a testament of the quality of our
  Open-Platypus dataset, and opens opportunities for more improvements in the
  field. Project page: https://platypus-llm.github.io
paper_authors._id: 64dacfddc8d191a54ac571a9
paper_authors.name: Ariel N. Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-15T01:28:39.736Z
paper_authors._id: 638bcfa91987d67b340e6c1c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
paper_authors.fullname: Ariel N. Lee
paper_authors.type: user
paper_authors.user: arielnlee
paper_authors._id: 64dacfddc8d191a54ac571aa
paper_authors.name: Cole J. Hunter
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-08-15T01:07:41.915Z
paper_authors._id: 638c6af459829706655239d2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638c6af459829706655239d2/6T2NsJGxkrMivVFbF93US.jpeg
paper_authors.fullname: lilloukas
paper_authors.type: user
paper_authors.user: lilloukas
paper_authors._id: 64dacfddc8d191a54ac571ab
paper_authors.name: Nataniel Ruiz
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-15T14:54:20.721Z
paper_authors._id: 6318c6e6cb116eab31d70a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
paper_authors.fullname: Nataniel
paper_authors.type: user
paper_authors.user: flavoredquark
paper_title: Platypus: Quick, Cheap, and Powerful Refinement of LLMs
paper_summary: We present Platypus, a family of fine-tuned and merged Large
  Language Models (LLMs) that achieves the strongest performance and currently
  stands at first place in HuggingFace's Open LLM Leaderboard as of the release
  date of this work. In this work we describe (1) our curated dataset
  Open-Platypus, that is a subset of other open datasets and which
  we release to the public (2) our process of fine-tuning and merging
  LoRA modules in order to conserve the strong prior of pretrained LLMs, while
  bringing specific domain knowledge to the surface (3) our efforts in checking
  for test data leaks and contamination in the training data, which can inform
  future research. Specifically, the Platypus family achieves strong performance
  in quantitative LLM metrics across model sizes, topping the global Open LLM
  leaderboard while using just a fraction of the fine-tuning data and overall
  compute that are required for other state-of-the-art fine-tuned LLMs. In
  particular, a 13B Platypus model can be trained on a single A100 GPU
  using 25k questions in 5 hours. This is a testament of the quality of our
  Open-Platypus dataset, and opens opportunities for more improvements in the
  field. Project page: https://platypus-llm.github.io
paper_ai_keywords: fine-tuning merged Large Language Models (LLMs) Open LLM Leaderboard Open-Platypus dataset LoRA modules pretrained LLMs domain knowledge test data leaks quantitative LLM metrics global Open LLM leaderboard