🔥 77.2% on MMLU with 3.7B parameters 🚀
text
🔥 77.2% on MMLU with 3.7B parameters 🚀
new_line
new_line
... 3.7B active parameters, 40B in total parameters 📊
text
... 3.7B active parameters, 40B in total parameters 📊
new_line
new_line
7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
text
7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
new_line
new_line
Exciting enough? 😲
text
Exciting enough? 😲
new_line
new_line
That's Yuan2-M32 for you, released by IEIT-Yuan.
text
That's Yuan2-M32 for you, released by IEIT-Yuan.
new_line
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
text
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
new_line
new_line
32 experts with 2 active in generation ✌️
text
32 experts with 2 active in generation ✌️
new_line
new_line
8,192 context length 📝
text
8,192 context length 📝
new_line
new_line
Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
text
Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
new_line
new_line
Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
text
Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
new_line
new_line
Open-source - Apache 2.0 📜
text
Open-source - Apache 2.0 📜
new_line
new_line
Vocabulary size of 135,040 🗣️
text
Vocabulary size of 135,040 🗣️
new_line
new_line
Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
text
Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
new_line
new_line
Models: 
text
Models: 
https://cdn-avatars.huggingface.co/v1/production/uploads/652ca9dacaddc9e60829269d/fJVgMovghqqrbVouhr6kg.png
https://huggingface.co/IEITYuan
IEITYuan
org
resource
https://huggingface.co/IEITYuan
 🌐
text
 🌐
new_line
Paper: 
text
Paper: 
Yuan 2.0-M32: Mixture of Experts with Attention Router (2405.17976)
https://huggingface.co/papers/2405.17976
2405.17976
paper
resource
https://huggingface.co/papers/2405.17976
 📄
text
 📄
🔥 77.2% on MMLU with 3.7B parameters 🚀

... 3.7B active parameters, 40B in total parameters 📊

7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉

Exciting enough? 😲

That's Yuan2-M32 for you, released by IEIT-Yuan.
A new 40B Mixture of Experts using a new Attention Router mechanism 🧠

32 experts with 2 active in generation ✌️

8,192 context length 📝

Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.

Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.

Open-source - Apache 2.0 📜

Vocabulary size of 135,040 🗣️

Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸

Models: https://huggingface.co/IEITYuan 🌐
Paper: https://huggingface.co/papers/2405.17976 📄