gradientai
/

Llama-3-8B-Instruct-Gradient-4194k

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

Update README.md

#3

by forrest-gradient - opened May 4

base: refs/heads/main

←

from: refs/pr/3

Discussion Files changed

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -42,7 +42,7 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | Initialize From        | LLaMA-3 8B| 65K       | 262K      | 524k      | 1048k     |
 | Sequence Length 2^N    | 16        | 18        | 19        | 20        | 22        |
 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
-| Batch Size             | 1         | 1         | 8         | 8         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
 | Steps                  | 30        | 24        | 50        | 50        | 11        |
 | Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 184549376 |

 | Initialize From        | LLaMA-3 8B| 65K       | 262K      | 524k      | 1048k     |
 | Sequence Length 2^N    | 16        | 18        | 19        | 20        | 22        |
 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
+| Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
 | Steps                  | 30        | 24        | 50        | 50        | 11        |
 | Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 184549376 |