gradientai
/

Llama-3-8B-Instruct-Gradient-4194k

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

forrest-gradient commited on May 4

Commit

69135eb

•

1 Parent(s): b738f0c

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -44,8 +44,8 @@ For training data, we generate long contexts by augmenting [SlimPajama](https://
 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
 | Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
-| Steps                  | 30        | 24        | 50        | 50        | 11        |
-| Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 184549376 |
 | Learning Rate          | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  |
 | # GPUs                 | 8         | 32        | 512       | 512       | 512       |
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |

 | RoPE theta             | 15.3 M    | 207.1 M   | 1.06B     | 2.80B     | 45.2B     |
 | Batch Size             | 1         | 1         | 16         | 16         | 2         |
 | Gradient Accumulation Steps | 32    | 16        | 1         | 1         | 2         |
+| Steps                  | 30        | 24        | 50        | 50        | 12        |
+| Total Tokens           | 62914560  | 100663296 | 419430400 | 838860800 | 201326592 |
 | Learning Rate          | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  | 2.00E-05  |
 | # GPUs                 | 8         | 32        | 512       | 512       | 512       |
 | GPU Type               | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S | NVIDIA L40S |