title: JetMoE: Reaching Llama2 Performance with 0.1M Dollars
summary: Large Language Models (LLMs) have achieved remarkable results, but their
  increasing resource demand has become a major obstacle to the development of
  powerful and accessible super-human intelligence. This report introduces
  JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
  from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
  low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
  outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
  Llama2-13B-Chat model. These results suggest that LLM training can be much more
  cost-effective than generally thought. JetMoE-8B is based on an efficient
  Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
  and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
  to have 8B parameters while only activating 2B for each input token, reducing
  inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
  is highly open and academia-friendly, using only public datasets and training
  code. All training parameters and data mixtures have been detailed in this
  report to facilitate future efforts in the development of open foundation
  models. This transparency aims to encourage collaboration and further
  advancements in the field of accessible and efficient LLMs. The model weights
  are publicly available at https://github.com/myshell-ai/JetMoE.
paper_authors._id: 66188b1112e9f8ef47c1bacf
paper_authors.name: Yikang Shen
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-06-25T18:40:33.328Z
paper_authors._id: 6463995939359568c63d740c
paper_authors.avatarUrl: /avatars/11caacefc68c1259dbc8764cc4340481.svg
paper_authors.fullname: Yikang Shen
paper_authors.type: user
paper_authors.user: YikangS
paper_authors._id: 66188b1112e9f8ef47c1bad0
paper_authors.name: Zhen Guo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T03:05:05.402Z
paper_authors._id: 638e4e66629b4d0a62ce1bf3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638e4e66629b4d0a62ce1bf3/s7uQ2qmee2CaXfjZDOovZ.jpeg
paper_authors.fullname: Zhen Guo
paper_authors.type: user
paper_authors.user: zguo0525
paper_authors._id: 66188b1112e9f8ef47c1bad1
paper_authors.name: Tianle Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:09:49.339Z
paper_authors._id: 64459357c525660aa20be337
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/1fryBXRuSvAgMIDkpWBpI.jpeg
paper_authors.fullname: Tianle Cai
paper_authors.type: user
paper_authors.user: tianlecai
paper_authors._id: 66188b1112e9f8ef47c1bad2
paper_authors.name: Zengyi Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:09:55.455Z
paper_authors._id: 641de0213239b631552713e4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641de0213239b631552713e4/nRul74VIwPqVOhvzcBfVH.jpeg
paper_authors.fullname: Zengyi Qin
paper_authors.type: user
paper_authors.user: Zengyi-Qin
paper_title: JetMoE: Reaching Llama2 Performance with 0.1M Dollars
paper_summary: Large Language Models (LLMs) have achieved remarkable results, but their
  increasing resource demand has become a major obstacle to the development of
  powerful and accessible super-human intelligence. This report introduces
  JetMoE-8B, a new LLM trained with less than $0.1 million, using 1.25T tokens
  from carefully mixed open-source corpora and 30,000 H100 GPU hours. Despite its
  low cost, the JetMoE-8B demonstrates impressive performance, with JetMoE-8B
  outperforming the Llama2-7B model and JetMoE-8B-Chat surpassing the
  Llama2-13B-Chat model. These results suggest that LLM training can be much more
  cost-effective than generally thought. JetMoE-8B is based on an efficient
  Sparsely-gated Mixture-of-Experts (SMoE) architecture, composed of attention
  and feedforward experts. Both layers are sparsely activated, allowing JetMoE-8B
  to have 8B parameters while only activating 2B for each input token, reducing
  inference computation by about 70% compared to Llama2-7B. Moreover, JetMoE-8B
  is highly open and academia-friendly, using only public datasets and training
  code. All training parameters and data mixtures have been detailed in this
  report to facilitate future efforts in the development of open foundation
  models. This transparency aims to encourage collaboration and further
  advancements in the field of accessible and efficient LLMs. The model weights
  are publicly available at https://github.com/myshell-ai/JetMoE.
paper_ai_summary: JetMoE-8B, a cost-effective large language model with 8 billion parameters, achieves impressive performance using a Sparsely-gated Mixture-of-Experts architecture, demonstrating efficient use of resources and computational savings.
paper_ai_keywords: Sparsely-gated Mixture-of-Experts SMoE attention experts feedforward experts