𝗛𝘂𝗻𝘆𝘂𝗮𝗻-𝗟𝗮𝗿𝗴𝗲 𝗷𝘂𝘀𝘁 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗯𝘆 𝗧𝗲𝗻𝗰𝗲𝗻𝘁: 𝗟𝗮𝗿𝗴𝗲𝘀𝘁 𝗲𝘃𝗲𝗿 𝗼𝗽𝗲𝗻 𝗠𝗼𝗘 𝗟𝗟𝗠, 𝗼𝗻𝗹𝘆 𝟱𝟮𝗕 𝗮𝗰𝘁𝗶𝘃𝗲 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀 𝗯𝘂𝘁 𝗯𝗲𝗮𝘁𝘀 𝗟𝗟𝗮𝗠𝗔 𝟯.𝟭-𝟰𝟬𝟱𝗕 𝗼𝗻 𝗺𝗼𝘀𝘁 𝗮𝗰𝗮𝗱𝗲𝗺𝗶𝗰 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 🚀
text
𝗛𝘂𝗻𝘆𝘂𝗮𝗻-𝗟𝗮𝗿𝗴𝗲 𝗷𝘂𝘀𝘁 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗯𝘆 𝗧𝗲𝗻𝗰𝗲𝗻𝘁: 𝗟𝗮𝗿𝗴𝗲𝘀𝘁 𝗲𝘃𝗲𝗿 𝗼𝗽𝗲𝗻 𝗠𝗼𝗘 𝗟𝗟𝗠, 𝗼𝗻𝗹𝘆 𝟱𝟮𝗕 𝗮𝗰𝘁𝗶𝘃𝗲 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀 𝗯𝘂𝘁 𝗯𝗲𝗮𝘁𝘀 𝗟𝗟𝗮𝗠𝗔 𝟯.𝟭-𝟰𝟬𝟱𝗕 𝗼𝗻 𝗺𝗼𝘀𝘁 𝗮𝗰𝗮𝗱𝗲𝗺𝗶𝗰 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 🚀
new_line
new_line
⚡ Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
text
⚡ Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input
new_line
new_line
🧪 Trained on 7T tokens, including 1.5T tokens of synthetic data
text
🧪 Trained on 7T tokens, including 1.5T tokens of synthetic data
new_line
new_line
🏗️ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
text
🏗️ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded
new_line
new_line
📊 Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
text
📊 Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
new_line
‣ Impressive perf on MATH: 77.4
text
‣ Impressive perf on MATH: 77.4
new_line
new_line
🐋 Large context length: up to 256K tokens
text
🐋 Large context length: up to 256K tokens
new_line
new_line
🔒 License:
text
🔒 License:
new_line
‣ Commercial use allowed, except if your products have >100M monthly active users
text
‣ Commercial use allowed, except if your products have >100M monthly active users
new_line
‣ No access in the EU
text
‣ No access in the EU
new_line
new_line
🤗 Model weights available on HF!
text
🤗 Model weights available on HF!
new_line
new_line
Read the full paper here 👉 
text
Read the full paper here 👉 
Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated
  Parameters by Tencent (2411.02265)
https://huggingface.co/papers/2411.02265
2411.02265
paper
resource
https://huggingface.co/papers/2411.02265
𝗛𝘂𝗻𝘆𝘂𝗮𝗻-𝗟𝗮𝗿𝗴𝗲 𝗷𝘂𝘀𝘁 𝗿𝗲𝗹𝗲𝗮𝘀𝗲𝗱 𝗯𝘆 𝗧𝗲𝗻𝗰𝗲𝗻𝘁: 𝗟𝗮𝗿𝗴𝗲𝘀𝘁 𝗲𝘃𝗲𝗿 𝗼𝗽𝗲𝗻 𝗠𝗼𝗘 𝗟𝗟𝗠, 𝗼𝗻𝗹𝘆 𝟱𝟮𝗕 𝗮𝗰𝘁𝗶𝘃𝗲 𝗽𝗮𝗿𝗮𝗺𝗲𝘁𝗲𝗿𝘀 𝗯𝘂𝘁 𝗯𝗲𝗮𝘁𝘀 𝗟𝗟𝗮𝗠𝗔 𝟯.𝟭-𝟰𝟬𝟱𝗕 𝗼𝗻 𝗺𝗼𝘀𝘁 𝗮𝗰𝗮𝗱𝗲𝗺𝗶𝗰 𝗯𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸𝘀 🚀

⚡ Mixture of Experts (MoE) architecture: 389 B parameters in total, but only 52B are activated for any input

🧪 Trained on 7T tokens, including 1.5T tokens of synthetic data

🏗️ Architecture : Novel "recycle routing" prevents token dropping when experts are overrloaded

📊 Great benchmark results: Surpasses Llama-3-405B-Instruct in most benchmarks although it has 8x fewer active parameters
‣ Impressive perf on MATH: 77.4

🐋 Large context length: up to 256K tokens

🔒 License:
‣ Commercial use allowed, except if your products have >100M monthly active users
‣ No access in the EU

🤗 Model weights available on HF!

Read the full paper here 👉 https://huggingface.co/papers/2411.02265
🔥
Hooo5 YaTharThShaRma999 John6666 Chief-Inspector
🚀
Hooo5 fdaudens YaTharThShaRma999 prithivMLmods
👍
Rsln