title: FLM-101B: An Open LLM and How to Train It with $100K Budget
summary: Large language models (LLMs) have achieved remarkable success in NLP and
  multimodal tasks. Despite these successes, their development faces two main
  challenges: (i) high computational cost; and (ii) difficulty in conducting fair
  and objective evaluations. LLMs are prohibitively expensive, making it feasible
  for only a few major players to undertake their training, thereby constraining
  both research and application opportunities. This underscores the importance of
  cost-effective LLM training. In this paper, we utilize a growth strategy to
  significantly reduce LLM training cost. We demonstrate that an LLM with 101B
  parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
  systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
  existing evaluations that focus more on knowledge-oriented abilities. We
  introduce our benchmark including evaluations on important aspects of
  intelligence including symbolic mapping, itrule understanding, pattern mining,
  and anti-interference. Such evaluations minimize the potential impact of
  memorization. Experimental results show that our model FLM-101B, trained with a
  budget of 100K, achieves comparable performance to powerful and well-known
  models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
  contexts unseen in training data. The checkpoint of FLM-101B will be
  open-sourced at https://huggingface.co/CofeAI/FLM-101B.
paper_authors._id: 64fa7f49b961d0d12c6d4592
paper_authors.name: Xiang Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T10:52:23.180Z
paper_authors._id: 64f593ebd08cbea2603a13a1
paper_authors.avatarUrl: /avatars/36f37a23372510e335097df058f06dc2.svg
paper_authors.fullname: XiangLi
paper_authors.type: user
paper_authors.user: XiangLi2022CASIA
paper_authors._id: 64fa7f49b961d0d12c6d4593
paper_authors.name: Yiqun Yao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T10:53:03.301Z
paper_authors._id: 64f9b61f0c7dbab99d9b0c16
paper_authors.avatarUrl: /avatars/fe5e09578cdebc97c970e6795c836a49.svg
paper_authors.fullname: Yiqun Yao
paper_authors.type: user
paper_authors.user: yqyao
paper_authors._id: 64fa7f49b961d0d12c6d4594
paper_authors.name: Xin Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T09:32:16.147Z
paper_authors._id: 647415007afa69c3c7a98f1f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
paper_authors.fullname: Xin Jiang
paper_authors.type: user
paper_authors.user: horiz94
paper_authors._id: 64fa7f49b961d0d12c6d4595
paper_authors.name: Xuezhi Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-08T19:31:58.927Z
paper_authors._id: 63c4f5588d95a5c770700ff8
paper_authors.avatarUrl: /avatars/4f64029f74cba179d263c0e4f1887427.svg
paper_authors.fullname: Jason Fang
paper_authors.type: user
paper_authors.user: jasonfang3900
paper_authors._id: 64fa7f49b961d0d12c6d4596
paper_authors.name: Xuying Meng
paper_authors._id: 64fa7f49b961d0d12c6d4597
paper_authors.name: Siqi Fan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-08T11:12:36.754Z
paper_authors._id: 64eaf68e1d8b651a9af2b306
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64eaf68e1d8b651a9af2b306/v7_rqGMY76rXtV0MYxrWG.jpeg
paper_authors.fullname: Siqi Fan
paper_authors.type: user
paper_authors.user: leofansq
paper_authors._id: 64fa7f49b961d0d12c6d4598
paper_authors.name: Peng Han
paper_authors._id: 64fa7f49b961d0d12c6d4599
paper_authors.name: Jing Li
paper_authors._id: 64fa7f49b961d0d12c6d459a
paper_authors.name: Li Du
paper_authors._id: 64fa7f49b961d0d12c6d459b
paper_authors.name: Bowen Qin
paper_authors._id: 64fa7f49b961d0d12c6d459c
paper_authors.name: Zheng Zhang
paper_authors._id: 64fa7f49b961d0d12c6d459d
paper_authors.name: Aixin Sun
paper_authors._id: 64fa7f49b961d0d12c6d459e
paper_authors.name: Yequan Wang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-09-08T01:56:26.361Z
paper_authors._id: 64f6db81867bbeac42b22b5d
paper_authors.avatarUrl: /avatars/cb945f3fce8f3b77f0316c83b1496a29.svg
paper_authors.fullname: Wang Yequan
paper_authors.type: user
paper_authors.user: thuwyq
paper_title: FLM-101B: An Open LLM and How to Train It with $100K Budget
paper_summary: Large language models (LLMs) have achieved remarkable success in NLP and
  multimodal tasks. Despite these successes, their development faces two main
  challenges: (i) high computational cost; and (ii) difficulty in conducting fair
  and objective evaluations. LLMs are prohibitively expensive, making it feasible
  for only a few major players to undertake their training, thereby constraining
  both research and application opportunities. This underscores the importance of
  cost-effective LLM training. In this paper, we utilize a growth strategy to
  significantly reduce LLM training cost. We demonstrate that an LLM with 101B
  parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
  systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
  existing evaluations that focus more on knowledge-oriented abilities. We
  introduce our benchmark including evaluations on important aspects of
  intelligence including symbolic mapping, itrule understanding, pattern mining,
  and anti-interference. Such evaluations minimize the potential impact of
  memorization. Experimental results show that our model FLM-101B, trained with a
  budget of 100K, achieves comparable performance to powerful and well-known
  models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
  contexts unseen in training data. The checkpoint of FLM-101B will be
  open-sourced at https://huggingface.co/CofeAI/FLM-101B.
paper_ai_keywords: parameter-efficient fine-tuning growth strategy denoising autoencoders (indirectly implied by the context of efficient training) symbolic mapping rule understanding pattern mining inter-interference IQ evaluation cost-effective LLM training