Commit
•
0d02073
1
Parent(s):
c7bf7e6
Update README.md
Browse files
README.md
CHANGED
|
@@ -17,7 +17,7 @@ For more info see our [End-to-end development service for custom LLMs and AI sys
|
|
| 17 |
|
| 18 |
This model extends LLama-3 8B's context length from 8k to 4194K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. For this stage, we trained on 185M tokens for this stage, and 1.6B tokens total for all stages, which is ~ 0.01% of Llama-3's original pre-training data.
|
| 19 |
|
| 20 |
-
**UPDATE WITH NIAH EVAL PLOT**
|
| 21 |
|
| 22 |
**Approach:**
|
| 23 |
|
|
@@ -54,7 +54,7 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
|
|
| 54 |
|
| 55 |
**Evaluation Details:**
|
| 56 |
|
| 57 |
-
**
|
| 58 |
|
| 59 |
```
|
| 60 |
EVAL_MAX_CONTEXT_LENGTH=1040200
|
|
|
|
| 17 |
|
| 18 |
This model extends LLama-3 8B's context length from 8k to 4194K, developed by Gradient, sponsored by compute from [Crusoe Energy](https://huggingface.co/crusoeai). It demonstrates that SOTA LLMs can learn to operate on long context with minimal training by appropriately adjusting RoPE theta. For this stage, we trained on 185M tokens for this stage, and 1.6B tokens total for all stages, which is ~ 0.01% of Llama-3's original pre-training data.
|
| 19 |
|
| 20 |
+
**[UPDATE WITH NIAH EVAL PLOT]**
|
| 21 |
|
| 22 |
**Approach:**
|
| 23 |
|
|
|
|
| 54 |
|
| 55 |
**Evaluation Details:**
|
| 56 |
|
| 57 |
+
**[UPDATE THESE NUMBERS]**
|
| 58 |
|
| 59 |
```
|
| 60 |
EVAL_MAX_CONTEXT_LENGTH=1040200
|