Introducing Zephyr 141B-A35B 🪁: 
text
Introducing Zephyr 141B-A35B 🪁: 
new_line
new_line
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
model
resource
https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1
new_line
new_line
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: 
text
Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: 
https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1
mistral-community/Mixtral-8x22B-v0.1
model
resource
https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1
new_line
new_line
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
text
Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:
new_line
new_line
🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by 
text
🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by 
@JW17
mention
JW17
 and 
text
 and 
@nlee-208
mention
nlee-208
 and 
text
 and 
@j6mes
mention
j6mes
 and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 
text
 and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 
new_line
new_line
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from 
text
🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from 
@LDJnr
mention
LDJnr
text
https://huggingface.co/datasets/LDJnr/Capybara
LDJnr/Capybara
dataset
resource
https://huggingface.co/datasets/LDJnr/Capybara
 and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
text
 and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.
new_line
new_line
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
text
What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.
new_line
new_line
Kudos to 
text
Kudos to 
@alvarobartt
mention
alvarobartt
text
@JW17
mention
JW17
 and 
text
 and 
@nlee-208
mention
nlee-208
 for this very nice and fast-paced collab!
text
 for this very nice and fast-paced collab!
new_line
new_line
For more details on the paper and dataset, checkout our collection: 
text
For more details on the paper and dataset, checkout our collection: 
https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
collection
resource
https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
Introducing Zephyr 141B-A35B 🪁: 

https://huggingface.co/HuggingFaceH4/zephyr-orpo-141b-A35b-v0.1

Yesterday, Mistral released their latest base model (via magnet link of course 😅) and the community quickly converted it to transformers format and pushed it to the Hub: https://huggingface.co/mistral-community/Mixtral-8x22B-v0.1

Early evals of this model looked extremely strong, so we teamed up with Argilla and KAIST AI to cook up a Zephyr recipe with a few new alignment techniques that came out recently:

🧑‍🍳 Align the base model with Odds Ratio Preference Optimisation (ORPO). This novel algorithm developed by @JW17 and @nlee-208 and @j6mes and does not require an SFT step to achieve high performance and is thus much more computationally efficient than methods like DPO and PPO. 

🦫 Use a brand new dataset of 7k high-quality, multi-turn preferences that has been developed by our friends at Argilla. To create this dataset, they took the excellent Capybara SFT dataset from @LDJnr  https://huggingface.co/datasets/LDJnr/Capybara and converted it into a preference dataset by augmenting the final turn with responses from new LLMs that were then ranked by GPT-4.

What we find especially neat about this approach is that training on 7k samples only takes ~1.3h on 4 H100 nodes, yet produces a model that is very strong on chat benchmarks like IFEval and BBH.

Kudos to @alvarobartt @JW17 and @nlee-208 for this very nice and fast-paced collab!

For more details on the paper and dataset, checkout our collection: https://huggingface.co/collections/HuggingFaceH4/zephyr-orpo-6617eba2c5c0e2cc3c151524
❤️
alvarobartt mehungrymunchie nlee-208 SilentWraith JW17 samusenps hakunamatata1997 dakshdeephere NouRed victor osanseviero avinash02 fynnkroeger dillfrescott clem philschmid srisree rafaelsandroni not-lain netynet
🤗
beta3 clem SilentWraith JW17 mathiasn1 victor osanseviero NickyNicky alvarobartt dillfrescott philschmid not-lain