amanrangapur commited on
Commit
8aeed46
·
verified ·
1 Parent(s): bbc26e3

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -159,11 +159,11 @@ Core model results for OLMo 2 32B are found below.
159
 
160
  ## Model Details
161
 
162
- ### Pretraining
163
  | | **OLMo 2 32B** | **OLMo 2 13B** | **OLMo 2 7B** |
164
  |-------------------|------------|------------|------------|
165
  | Pretraining Stage 1 | 6 trillion tokens<br>(1.5 epoch) | 5 trillion tokens<br>(1.2 epochs) | 4 trillion tokens<br>(1 epoch) |
166
- | Pretraining Stage 2 | 100B tokens (2 runs)<br>300B tokens (1 run)<br>*merged* | 100B tokens (3 runs)<br>300B tokens (1 run)<br>*merged* | 50B tokens (3 runs)<br>*merged* |
167
  | Post-training | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-32b-pref-mix-v1)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-13b-preference-mix)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix)) |
168
 
169
  #### Stage 1: Initial Pretraining
@@ -171,7 +171,7 @@ Core model results for OLMo 2 32B are found below.
171
  - Coverage: 95%+ of total pretraining budget
172
  - 32B Model: ~1.5 epoch
173
 
174
- #### Stage 2: Fine-tuning
175
  - Dataset: Dolmino-Mix-1124
176
  - Two training mixes:
177
  - 100B tokens
 
159
 
160
  ## Model Details
161
 
162
+ ### Training
163
  | | **OLMo 2 32B** | **OLMo 2 13B** | **OLMo 2 7B** |
164
  |-------------------|------------|------------|------------|
165
  | Pretraining Stage 1 | 6 trillion tokens<br>(1.5 epoch) | 5 trillion tokens<br>(1.2 epochs) | 4 trillion tokens<br>(1 epoch) |
166
+ | Pretraining Stage 2 | 100B tokens (3 runs)<br>300B tokens (1 run)<br>*merged* | 100B tokens (3 runs)<br>300B tokens (1 run)<br>*merged* | 50B tokens (3 runs)<br>*merged* |
167
  | Post-training | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-32b-pref-mix-v1)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-13b-preference-mix)) | SFT + DPO + PPO<br>([preference mix](https://huggingface.co/datasets/allenai/olmo-2-1124-7b-preference-mix)) |
168
 
169
  #### Stage 1: Initial Pretraining
 
171
  - Coverage: 95%+ of total pretraining budget
172
  - 32B Model: ~1.5 epoch
173
 
174
+ #### Stage 2: Mid-training
175
  - Dataset: Dolmino-Mix-1124
176
  - Two training mixes:
177
  - 100B tokens