FLM-101B: An Open LLM and How to Train It with $100K Budget
Large language models (LLMs) have achieved remarkable success in NLP and
multimodal tasks. Despite these successes, their development faces two main
challenges: (i) high computational cost; and (ii) difficulty in conducting fair
and objective evaluations. LLMs are prohibitively expensive, making it feasible
for only a few major players to undertake their training, thereby constraining
both research and application opportunities. This underscores the importance of
cost-effective LLM training. In this paper, we utilize a growth strategy to
significantly reduce LLM training cost. We demonstrate that an LLM with 101B
parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
existing evaluations that focus more on knowledge-oriented abilities. We
introduce our benchmark including evaluations on important aspects of
intelligence including symbolic mapping, itrule understanding, pattern mining,
and anti-interference. Such evaluations minimize the potential impact of
memorization. Experimental results show that our model FLM-101B, trained with a
budget of 100K, achieves comparable performance to powerful and well-known
models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
contexts unseen in training data. The checkpoint of FLM-101B will be
open-sourced at https://huggingface.co/CofeAI/FLM-101B.
64fa7f49b961d0d12c6d4592
Xiang Li
admin_assigned
2023-09-08T10:52:23.180Z
64f593ebd08cbea2603a13a1
/avatars/36f37a23372510e335097df058f06dc2.svg
XiangLi
user
XiangLi2022CASIA
64fa7f49b961d0d12c6d4593
Yiqun Yao
admin_assigned
2023-09-08T10:53:03.301Z
64f9b61f0c7dbab99d9b0c16
/avatars/fe5e09578cdebc97c970e6795c836a49.svg
Yiqun Yao
user
yqyao
64fa7f49b961d0d12c6d4594
Xin Jiang
claimed_verified
2023-09-26T09:32:16.147Z
647415007afa69c3c7a98f1f
https://cdn-avatars.huggingface.co/v1/production/uploads/647415007afa69c3c7a98f1f/pEl0PozmzNK8_PwUMiikd.jpeg
Xin Jiang
user
horiz94
64fa7f49b961d0d12c6d4595
Xuezhi Fang
claimed_verified
2023-09-08T19:31:58.927Z
63c4f5588d95a5c770700ff8
/avatars/4f64029f74cba179d263c0e4f1887427.svg
Jason Fang
user
jasonfang3900
64fa7f49b961d0d12c6d4596
Xuying Meng
64fa7f49b961d0d12c6d4597
Siqi Fan
admin_assigned
2023-09-08T11:12:36.754Z
64eaf68e1d8b651a9af2b306
https://cdn-avatars.huggingface.co/v1/production/uploads/64eaf68e1d8b651a9af2b306/v7_rqGMY76rXtV0MYxrWG.jpeg
Siqi Fan
user
leofansq
64fa7f49b961d0d12c6d4598
Peng Han
64fa7f49b961d0d12c6d4599
Jing Li
64fa7f49b961d0d12c6d459a
Li Du
64fa7f49b961d0d12c6d459b
Bowen Qin
64fa7f49b961d0d12c6d459c
Zheng Zhang
64fa7f49b961d0d12c6d459d
Aixin Sun
64fa7f49b961d0d12c6d459e
Yequan Wang
extracted_pending
2023-09-08T01:56:26.361Z
64f6db81867bbeac42b22b5d
/avatars/cb945f3fce8f3b77f0316c83b1496a29.svg
Wang Yequan
user
thuwyq
FLM-101B: An Open LLM and How to Train It with $100K Budget
Large language models (LLMs) have achieved remarkable success in NLP and
multimodal tasks. Despite these successes, their development faces two main
challenges: (i) high computational cost; and (ii) difficulty in conducting fair
and objective evaluations. LLMs are prohibitively expensive, making it feasible
for only a few major players to undertake their training, thereby constraining
both research and application opportunities. This underscores the importance of
cost-effective LLM training. In this paper, we utilize a growth strategy to
significantly reduce LLM training cost. We demonstrate that an LLM with 101B
parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a
systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to
existing evaluations that focus more on knowledge-oriented abilities. We
introduce our benchmark including evaluations on important aspects of
intelligence including symbolic mapping, itrule understanding, pattern mining,
and anti-interference. Such evaluations minimize the potential impact of
memorization. Experimental results show that our model FLM-101B, trained with a
budget of 100K, achieves comparable performance to powerful and well-known
models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with
contexts unseen in training data. The checkpoint of FLM-101B will be
open-sourced at https://huggingface.co/CofeAI/FLM-101B.
A growth strategy reduces the cost of training large language models while achieving comparable performance, with a focus on evaluating IQ and minimizing memorization.
growth strategy IQ evaluation symbolic mapping itrule understanding pattern mining anti-interference