Platypus: Quick, Cheap, and Powerful Refinement of LLMs
We present Platypus, a family of fine-tuned and merged Large
Language Models (LLMs) that achieves the strongest performance and currently
stands at first place in HuggingFace's Open LLM Leaderboard as of the release
date of this work. In this work we describe (1) our curated dataset
Open-Platypus, that is a subset of other open datasets and which
we release to the public (2) our process of fine-tuning and merging
LoRA modules in order to conserve the strong prior of pretrained LLMs, while
bringing specific domain knowledge to the surface (3) our efforts in checking
for test data leaks and contamination in the training data, which can inform
future research. Specifically, the Platypus family achieves strong performance
in quantitative LLM metrics across model sizes, topping the global Open LLM
leaderboard while using just a fraction of the fine-tuning data and overall
compute that are required for other state-of-the-art fine-tuned LLMs. In
particular, a 13B Platypus model can be trained on a single A100 GPU
using 25k questions in 5 hours. This is a testament of the quality of our
Open-Platypus dataset, and opens opportunities for more improvements in the
field. Project page: https://platypus-llm.github.io
64dacfddc8d191a54ac571a9
Ariel N. Lee
extracted_confirmed
2023-08-15T01:28:39.736Z
638bcfa91987d67b340e6c1c
https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
Ariel N. Lee
user
arielnlee
64dacfddc8d191a54ac571aa
Cole J. Hunter
extracted_pending
2023-08-15T01:07:41.915Z
638c6af459829706655239d2
https://cdn-avatars.huggingface.co/v1/production/uploads/638c6af459829706655239d2/6T2NsJGxkrMivVFbF93US.jpeg
lilloukas
user
lilloukas
64dacfddc8d191a54ac571ab
Nataniel Ruiz
extracted_confirmed
2023-08-15T14:54:20.721Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
Platypus: Quick, Cheap, and Powerful Refinement of LLMs
We present Platypus, a family of fine-tuned and merged Large
Language Models (LLMs) that achieves the strongest performance and currently
stands at first place in HuggingFace's Open LLM Leaderboard as of the release
date of this work. In this work we describe (1) our curated dataset
Open-Platypus, that is a subset of other open datasets and which
we release to the public (2) our process of fine-tuning and merging
LoRA modules in order to conserve the strong prior of pretrained LLMs, while
bringing specific domain knowledge to the surface (3) our efforts in checking
for test data leaks and contamination in the training data, which can inform
future research. Specifically, the Platypus family achieves strong performance
in quantitative LLM metrics across model sizes, topping the global Open LLM
leaderboard while using just a fraction of the fine-tuning data and overall
compute that are required for other state-of-the-art fine-tuned LLMs. In
particular, a 13B Platypus model can be trained on a single A100 GPU
using 25k questions in 5 hours. This is a testament of the quality of our
Open-Platypus dataset, and opens opportunities for more improvements in the
field. Project page: https://platypus-llm.github.io
A fine-tuned and merged family of large language models named Platypus, using LoRA modules and a curated dataset, achieves top performance on the Open LLM Leaderboard with reduced data and compute.
Large Language Models LLMs fine-tuning Open LLM Leaderboard LoRA modules test data leaks training data contamination quantitative LLM metrics A100 GPU Open-Platypus dataset