title: Tele-FLM Technical Report
summary: Large language models (LLMs) have showcased profound capabilities in language
  understanding and generation, facilitating a wide array of applications.
  However, there is a notable paucity of detailed, open-sourced methodologies on
  efficiently scaling LLMs beyond 50 billion parameters with minimum
  trial-and-error cost and computational resources. In this report, we introduce
  Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
  features a stable, efficient pre-training paradigm and enhanced factual
  judgment capabilities. Tele-FLM demonstrates superior multilingual language
  modeling abilities, measured by BPB on textual corpus. Besides, in both English
  and Chinese foundation model evaluation, it is comparable to strong
  open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
  and DeepSeek-67B. In addition to the model weights, we share the core designs,
  engineering practices, and training details, which we expect to benefit both
  the academic and industrial communities.
paper_authors._id: 662b17cee5d910dd729834f9
paper_authors.name: Xiang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-26T10:05:34.081Z
paper_authors._id: 6486ea539264d020ddb2a06e
paper_authors.avatarUrl: /avatars/3d5f2e8e13c87ad89197f171395d9556.svg
paper_authors.fullname: Xiang Li
paper_authors.type: user
paper_authors.user: sleepylx
paper_authors._id: 662b17cee5d910dd729834fa
paper_authors.name: Yiqun Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T09:45:35.744Z
paper_authors._id: 64f9b61f0c7dbab99d9b0c16
paper_authors.avatarUrl: /avatars/fe5e09578cdebc97c970e6795c836a49.svg
paper_authors.fullname: Yiqun Yao
paper_authors.type: user
paper_authors.user: yqyao
paper_authors._id: 662b17cee5d910dd729834fb
paper_authors.name: Xin Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T09:45:55.068Z
paper_authors._id: 647415007afa69c3c7a98f1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
paper_authors.fullname: Xin Jiang
paper_authors.type: user
paper_authors.user: horiz94
paper_authors._id: 662b17cee5d910dd729834fc
paper_authors.name: Xuezhi Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:30:11.025Z
paper_authors._id: 63c4f5588d95a5c770700ff8
paper_authors.avatarUrl: /avatars/4f64029f74cba179d263c0e4f1887427.svg
paper_authors.fullname: Jason Fang
paper_authors.type: user
paper_authors.user: jasonfang3900
paper_authors._id: 662b17cee5d910dd729834fd
paper_authors.name: Chao Wang
paper_authors._id: 662b17cee5d910dd729834fe
paper_authors.name: Xinzhang Liu
paper_authors._id: 662b17cee5d910dd729834ff
paper_authors.name: Zihan Wang
paper_authors._id: 662b17cee5d910dd72983500
paper_authors.name: Yu Zhao
paper_authors._id: 662b17cee5d910dd72983501
paper_authors.name: Xin Wang
paper_authors._id: 662b17cee5d910dd72983502
paper_authors.name: Yuyao Huang
paper_authors._id: 662b17cee5d910dd72983503
paper_authors.name: Shuangyong Song
paper_authors._id: 662b17cee5d910dd72983504
paper_authors.name: Yongxiang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:03:54.318Z
paper_authors._id: 660b9ef3f4ab651901147194
paper_authors.avatarUrl: /avatars/12720ddf21084f9ec48313d156918d63.svg
paper_authors.fullname: yongxiang
paper_authors.type: user
paper_authors.user: liyongxiang
paper_authors._id: 662b17cee5d910dd72983505
paper_authors.name: Zheng Zhang
paper_authors._id: 662b17cee5d910dd72983506
paper_authors.name: Bo Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:06:14.593Z
paper_authors._id: 64acafac1aee69ece03af05a
paper_authors.avatarUrl: /avatars/6380777852546dbd432fed5544d5db0b.svg
paper_authors.fullname: Bo Zhao
paper_authors.type: user
paper_authors.user: BoZhaoHuggingFace
paper_authors._id: 662b17cee5d910dd72983507
paper_authors.name: Aixin Sun
paper_authors._id: 662b17cee5d910dd72983508
paper_authors.name: Yequan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-26T10:06:38.480Z
paper_authors._id: 64f6db81867bbeac42b22b5d
paper_authors.avatarUrl: /avatars/cb945f3fce8f3b77f0316c83b1496a29.svg
paper_authors.fullname: Wang Yequan
paper_authors.type: user
paper_authors.user: thuwyq
paper_authors._id: 662b17cee5d910dd72983509
paper_authors.name: Zhongjiang He
paper_authors._id: 662b17cee5d910dd7298350a
paper_authors.name: Zhongyuan Wang
paper_authors._id: 662b17cee5d910dd7298350b
paper_authors.name: Xuelong Li
paper_authors._id: 662b17cee5d910dd7298350c
paper_authors.name: Tiejun Huang
paper_title: Tele-FLM Technical Report
paper_summary: Large language models (LLMs) have showcased profound capabilities in language
  understanding and generation, facilitating a wide array of applications.
  However, there is a notable paucity of detailed, open-sourced methodologies on
  efficiently scaling LLMs beyond 50 billion parameters with minimum
  trial-and-error cost and computational resources. In this report, we introduce
  Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that
  features a stable, efficient pre-training paradigm and enhanced factual
  judgment capabilities. Tele-FLM demonstrates superior multilingual language
  modeling abilities, measured by BPB on textual corpus. Besides, in both English
  and Chinese foundation model evaluation, it is comparable to strong
  open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B
  and DeepSeek-67B. In addition to the model weights, we share the core designs,
  engineering practices, and training details, which we expect to benefit both
  the academic and industrial communities.
paper_ai_keywords: large language models (LLMs) pre-training paradigm BPB (bits per byte) textual corpus English foundation model Chinese foundation model Llama2-70B DeepSeek-67B