title: GEB-1.3B: Open Lightweight Large Language Model
summary: Recently developed large language models (LLMs) such as ChatGPT, Claude, and
  Llama have demonstrated impressive abilities, and even surpass human-level
  performance in several tasks. Despite their success, the resource-intensive
  demands of these models, requiring significant computational power for both
  training and inference, limit their deployment to high-performance servers.
  Additionally, the extensive calculation requirements of the models often lead
  to increased latency in response times. With the increasing need for LLMs to
  operate efficiently on CPUs, research about lightweight models that are
  optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a
  lightweight LLM trained on 550 billion tokens in both Chinese and English
  languages. We employ novel training techniques, including ROPE,
  Group-Query-Attention, and FlashAttention-2, to accelerate training while
  maintaining model performance. Additionally, we fine-tune the model using 10
  million samples of instruction data to enhance alignment. GEB-1.3B exhibits
  outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,
  outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.
  Notably, the FP32 version of GEB-1.3B achieves commendable inference times on
  CPUs, with ongoing efforts to further enhance speed through advanced
  quantization techniques. The release of GEB-1.3B as an open-source model marks
  a significant contribution to the development of lightweight LLMs, promising to
  foster further research and innovation in the field.
paper_authors._id: 666ff529b07525f0bdec8aee
paper_authors.name: Jie Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:13.898Z
paper_authors._id: 6593be1ba78a2778033e1e1a
paper_authors.avatarUrl: /avatars/f1b18cd1b4aa8f255b3b7f2a15cf08cf.svg
paper_authors.fullname: Wujie
paper_authors.type: user
paper_authors.user: Wu416732042
paper_authors._id: 666ff529b07525f0bdec8aef
paper_authors.name: Yufeng Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:15:53.783Z
paper_authors._id: 63464df0fe134dfd7a11f61d
paper_authors.avatarUrl: /avatars/d880f1c66945e91db7336f50f5dd7e41.svg
paper_authors.fullname: zhu
paper_authors.type: user
paper_authors.user: yufengzhu
paper_authors._id: 666ff529b07525f0bdec8af0
paper_authors.name: Lei Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:28:44.698Z
paper_authors._id: 6454d2ee1a543cf97b1ba671
paper_authors.avatarUrl: /avatars/30edc30cf3684bf685aabffcec8a3c30.svg
paper_authors.fullname: shen
paper_authors.type: user
paper_authors.user: lorashen
paper_authors._id: 666ff529b07525f0bdec8af1
paper_authors.name: Xuqing Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-17T14:16:00.236Z
paper_authors._id: 666a94e8cf27eed0f226cf2d
paper_authors.avatarUrl: /avatars/282f548c26c62660c9cd6b8d806ebe92.svg
paper_authors.fullname: xuqing lu
paper_authors.type: user
paper_authors.user: luxq
paper_title: GEB-1.3B: Open Lightweight Large Language Model
paper_summary: Recently developed large language models (LLMs) such as ChatGPT, Claude, and
  Llama have demonstrated impressive abilities, and even surpass human-level
  performance in several tasks. Despite their success, the resource-intensive
  demands of these models, requiring significant computational power for both
  training and inference, limit their deployment to high-performance servers.
  Additionally, the extensive calculation requirements of the models often lead
  to increased latency in response times. With the increasing need for LLMs to
  operate efficiently on CPUs, research about lightweight models that are
  optimized for CPU inference has emerged. In this work, we introduce GEB-1.3B, a
  lightweight LLM trained on 550 billion tokens in both Chinese and English
  languages. We employ novel training techniques, including ROPE,
  Group-Query-Attention, and FlashAttention-2, to accelerate training while
  maintaining model performance. Additionally, we fine-tune the model using 10
  million samples of instruction data to enhance alignment. GEB-1.3B exhibits
  outstanding performance on general benchmarks such as MMLU, C-Eval, and CMMLU,
  outperforming comparative models such as MindLLM-1.3B and TinyLLaMA-1.1B.
  Notably, the FP32 version of GEB-1.3B achieves commendable inference times on
  CPUs, with ongoing efforts to further enhance speed through advanced
  quantization techniques. The release of GEB-1.3B as an open-source model marks
  a significant contribution to the development of lightweight LLMs, promising to
  foster further research and innovation in the field.
paper_ai_summary: GEB-1.3B, a lightweight LLM trained on multilingual data, employs novel techniques to achieve efficient CPU inference and outperforms other lightweight models on benchmarks.
paper_ai_keywords: large language models LLMs ChatGPT Claude Llama GEB-1.3B lightweight models CPU inference ROPE Group-Query-Attention FlashAttention-2 fine-tuning MMLU C-Eval CMMLU MindLLM-1.3B TinyLLaMA-1.1B FP32 quantization techniques