Initial GPTQ model commit
Browse files
README.md
CHANGED
@@ -230,7 +230,7 @@ Thank you to all my generous patrons and donaters!
|
|
230 |
|
231 |
# OpenOrca x OpenChat - Preview2 - 13B
|
232 |
|
233 |
-
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing
|
234 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
235 |
|
236 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
@@ -242,7 +242,7 @@ As well, this is done with <1/10th the compute requirement and using <20% of the
|
|
242 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
243 |
|
244 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
245 |
-
We have utilized OpenChat
|
246 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
247 |
|
248 |
|
|
|
230 |
|
231 |
# OpenOrca x OpenChat - Preview2 - 13B
|
232 |
|
233 |
+
We have used our own [OpenOrca dataset](https://huggingface.co/datasets/Open-Orca/OpenOrca) to fine-tune Llama2-13B using [OpenChat](https://huggingface.co/openchat) packing.
|
234 |
This dataset is our attempt to reproduce the dataset generated for Microsoft Research's [Orca Paper](https://arxiv.org/abs/2306.02707).
|
235 |
|
236 |
This second preview release is trained on a curated filtered subset of most of our GPT-4 augmented data.
|
|
|
242 |
We have run extensive evaluations internally and expect this model to **place number 1** on both the HuggingFaceH4 Open LLM Leaderboard and the GPT4ALL Leaderboard for 13B models.
|
243 |
|
244 |
"One" of [OpenChat](https://huggingface.co/openchat) has joined our team, and we'd like to provide special thanks for their training of this model!
|
245 |
+
We have utilized OpenChat [MultiPack algorithm](https://github.com/imoneoi/multipack_sampler) which achieves 99.85% bin-packing efficiency on our dataset.
|
246 |
This has significantly reduced training time, with efficiency improvement of 3-10X over traditional methods.
|
247 |
|
248 |
|