Update README.md
Browse files
README.md
CHANGED
@@ -159,11 +159,11 @@ Core model results for OLMo 2 32B are found below.
|
|
159 |
|
160 |
## Model Details
|
161 |
|
162 |
-
###
|
163 |
| | **OLMo 2 32B** | **OLMo 2 13B** | **OLMo 2 7B** |
|
164 |
|-------------------|------------|------------|------------|
|
165 |
| Pretraining Stage 1 | 6 trillion tokens<br>(1.5 epoch) | 5 trillion tokens<br>(1.2 epochs) | 4 trillion tokens<br>(1 epoch) |
|
166 |
-
| Pretraining Stage 2 | 100B tokens (
|
167 |
| Post-training | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-32b-pref-mix-v1)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-13b-preference-mix)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix)) |
|
168 |
|
169 |
#### Stage 1: Initial Pretraining
|
@@ -171,7 +171,7 @@ Core model results for OLMo 2 32B are found below.
|
|
171 |
- Coverage: 95%+ of total pretraining budget
|
172 |
- 32B Model: ~1.5 epoch
|
173 |
|
174 |
-
#### Stage 2:
|
175 |
- Dataset: Dolmino-Mix-1124
|
176 |
- Two training mixes:
|
177 |
- 100B tokens
|
|
|
159 |
|
160 |
## Model Details
|
161 |
|
162 |
+
### Training
|
163 |
| | **OLMo 2 32B** | **OLMo 2 13B** | **OLMo 2 7B** |
|
164 |
|-------------------|------------|------------|------------|
|
165 |
| Pretraining Stage 1 | 6 trillion tokens<br>(1.5 epoch) | 5 trillion tokens<br>(1.2 epochs) | 4 trillion tokens<br>(1 epoch) |
|
166 |
+
| Pretraining Stage 2 | 100B tokens (3 runs)<br>300B tokens (1 run)<br>*merged* | 100B tokens (3 runs)<br>300B tokens (1 run)<br>*merged* | 50B tokens (3 runs)<br>*merged* |
|
167 |
| Post-training | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-32b-pref-mix-v1)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-13b-preference-mix)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix)) |
|
168 |
|
169 |
#### Stage 1: Initial Pretraining
|
|
|
171 |
- Coverage: 95%+ of total pretraining budget
|
172 |
- 32B Model: ~1.5 epoch
|
173 |
|
174 |
+
#### Stage 2: Mid-training
|
175 |
- Dataset: Dolmino-Mix-1124
|
176 |
- Two training mixes:
|
177 |
- 100B tokens
|