Update README.md
Browse files
README.md
CHANGED
|
@@ -16,7 +16,7 @@ datasets:
|
|
| 16 |
|
| 17 |
# OpenOrca x OpenChat - Preview2 - 13B
|
| 18 |
|
| 19 |
-
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing
|
| 20 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
| 21 |
|
| 22 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
|
@@ -28,7 +28,7 @@ As well, this is done with <1/10th the compute requirement and using <20% of the
|
|
| 28 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
| 29 |
|
| 30 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
| 31 |
-
We have utilized OpenChat
|
| 32 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
| 33 |
|
| 34 |
|
|
|
|
| 16 |
|
| 17 |
# OpenOrca x OpenChat - Preview2 - 13B
|
| 18 |
|
| 19 |
+
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing.
|
| 20 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
| 21 |
|
| 22 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
|
|
|
| 28 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
| 29 |
|
| 30 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
| 31 |
+
We have utilized OpenChat [MultiPack algorithm](https://github.com/imoneoi/multipack_sampler) which achieves 99.85% bin-packing efficiency on our dataset.
|
| 32 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
| 33 |
|
| 34 |
|