content.raw: 🔥 77.2% on MMLU with 3.7B parameters 🚀
content.type: text
content.value: 🔥 77.2% on MMLU with 3.7B parameters 🚀
content.type: new_line
content.type: new_line
content.raw: ... 3.7B active parameters, 40B in total parameters 📊
content.type: text
content.value: ... 3.7B active parameters, 40B in total parameters 📊
content.type: new_line
content.type: new_line
content.raw: 7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
content.type: text
content.value: 7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
content.type: new_line
content.type: new_line
content.raw: Exciting enough? 😲
content.type: text
content.value: Exciting enough? 😲
content.type: new_line
content.type: new_line
content.raw: That's Yuan2-M32 for you, released by IEIT-Yuan.
content.type: text
content.value: That's Yuan2-M32 for you, released by IEIT-Yuan.
content.type: new_line
content.raw: A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
content.type: text
content.value: A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
content.type: new_line
content.type: new_line
content.raw: 32 experts with 2 active in generation ✌️
content.type: text
content.value: 32 experts with 2 active in generation ✌️
content.type: new_line
content.type: new_line
content.raw: 8,192 context length 📝
content.type: text
content.value: 8,192 context length 📝
content.type: new_line
content.type: new_line
content.raw: Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
content.type: text
content.value: Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
content.type: new_line
content.type: new_line
content.raw: Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
content.type: text
content.value: Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
content.type: new_line
content.type: new_line
content.raw: Open-source - Apache 2.0 📜
content.type: text
content.value: Open-source - Apache 2.0 📜
content.type: new_line
content.type: new_line
content.raw: Vocabulary size of 135,040 🗣️
content.type: text
content.value: Vocabulary size of 135,040 🗣️
content.type: new_line
content.type: new_line
content.raw: Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
content.type: text
content.value: Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
content.type: new_line
content.type: new_line
content.raw: Models: 
content.type: text
content.value: Models: 
content.image: https://cdn-avatars.huggingface.co/v1/production/uploads/652ca9dacaddc9e60829269d/fJVgMovghqqrbVouhr6kg.png
content.raw: https://huggingface.co/IEITYuan
content.id: IEITYuan
content.type: org
content.type: resource
content.url: https://huggingface.co/IEITYuan
content.raw:  🌐
content.type: text
content.value:  🌐
content.type: new_line
content.raw: Paper: 
content.type: text
content.value: Paper: 
content.label: Yuan 2.0-M32: Mixture of Experts with Attention Router (2405.17976)
content.raw: https://huggingface.co/papers/2405.17976
content.id: 2405.17976
content.type: paper
content.type: resource
content.url: https://huggingface.co/papers/2405.17976
content.raw:  📄
content.type: text
content.value:  📄
rawContent: 🔥 77.2% on MMLU with 3.7B parameters 🚀
  
  ... 3.7B active parameters, 40B in total parameters 📊
  
  7.4 GFlops forward computation per token, 1/19 of Llama3-70B 📉
  
  Exciting enough? 😲
  
  That's Yuan2-M32 for you, released by IEIT-Yuan.
  A new 40B Mixture of Experts using a new Attention Router mechanism 🧠
  
  32 experts with 2 active in generation ✌️
  
  8,192 context length 📝
  
  Trained on 2T tokens, using 9.25% of the compute required by the dense models 🛠️.
  
  Yuan 2.0-M32 employs fine-tuning techniques to adjust to longer sequence lengths, utilizing a modified base value in the Rotary Position Embedding to maintain performance over extended contexts 🔄.
  
  Open-source - Apache 2.0 📜
  
  Vocabulary size of 135,040 🗣️
  
  Outperforms Mixtral 8x7B (47B total parameters, 12.9B active parameters) on all benchmarks and almost gives Llama 3 70B run for its money 💸
  
  Models: https://huggingface.co/IEITYuan 🌐
  Paper: https://huggingface.co/papers/2405.17976 📄