DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
The rapid development of open-source large language models (LLMs) has been
truly remarkable. However, the scaling law described in previous literature
presents varying conclusions, which casts a dark cloud over scaling LLMs. We
delve into the study of scaling laws and present our distinctive findings that
facilitate scaling of large scale models in two commonly used open-source
configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
LLM, a project dedicated to advancing open-source language models with a
long-term perspective. To support the pre-training phase, we have developed a
dataset that currently consists of 2 trillion tokens and is continuously
expanding. We further conduct supervised fine-tuning (SFT) and Direct
Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
creation of DeepSeek Chat models. Our evaluation results demonstrate that
DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
the domains of code, mathematics, and reasoning. Furthermore, open-ended
evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
compared to GPT-3.5.
659b5f4c2fe7ca485f422838
DeepSeek-AI
659b5f4c2fe7ca485f42283a
Xiao Bi
659b5f4c2fe7ca485f42283b
Deli Chen
659b5f4c2fe7ca485f42283c
Guanting Chen
659b5f4c2fe7ca485f42283d
Shanhuang Chen
659b5f4c2fe7ca485f42283e
Damai Dai
admin_assigned
2024-01-08T11:14:55.147Z
659389f8de82e1ef7b9a8b13
/avatars/896ed9f4cdbd317493b303d070b7e12a.svg
Damai Dai
user
DeepSeekDDM
659b5f4c2fe7ca485f42283f
Chengqi Deng
659b5f4c2fe7ca485f422840
Honghui Ding
659b5f4c2fe7ca485f422841
Kai Dong
659b5f4c2fe7ca485f422842
Qiushi Du
659b5f4c2fe7ca485f422843
Zhe Fu
659b5f4c2fe7ca485f422844
Huazuo Gao
659b5f4c2fe7ca485f422845
Kaige Gao
659b5f4c2fe7ca485f422846
Wenjun Gao
659b5f4c2fe7ca485f422847
Ruiqi Ge
659b5f4c2fe7ca485f422848
Kang Guan
659b5f4c2fe7ca485f422849
Daya Guo
admin_assigned
2024-01-08T11:22:02.420Z
653df20eaa1f487614da4db1
/avatars/12b27ce2c59f53b7e464039deab36a5d.svg
Daya Guo
user
guoday
659b5f4c2fe7ca485f42284a
Jianzhong Guo
659b5f4c2fe7ca485f42284b
Guangbo Hao
659b5f4c2fe7ca485f42284c
Zhewen Hao
659b5f4c2fe7ca485f42284d
Ying He
659b5f4c2fe7ca485f42284e
Wenjie Hu
659b5f4c2fe7ca485f42284f
Panpan Huang
659b5f4c2fe7ca485f422850
Erhang Li
659b5f4c2fe7ca485f422851
Guowei Li
659b5f4c2fe7ca485f422852
Jiashi Li
admin_assigned
2024-01-08T11:23:56.641Z
64fca5f28d50404bc42ca78a
/avatars/ae01ac0296d6ce1277dacb6894f570b8.svg
Jiashi Li
user
Beginlner
659b5f4c2fe7ca485f422853
Yao Li
659b5f4c2fe7ca485f422854
Y. K. Li
659b5f4c2fe7ca485f422855
Wenfeng Liang
659b5f4c2fe7ca485f422856
Fangyun Lin
659b5f4c2fe7ca485f422857
A. X. Liu
659b5f4c2fe7ca485f422858
Bo Liu
claimed_verified
2024-01-08T21:57:26.682Z
635e3a76106f984574c36409
https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
Bo Liu
user
Benjamin-eecs
659b5f4c2fe7ca485f422859
Wen Liu
claimed_verified
2024-01-22T10:31:39.208Z
63198e8802fb322037332f2d
/avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
Wen Liu
user
doubility123
659b5f4c2fe7ca485f42285a
Xiaodong Liu
659b5f4c2fe7ca485f42285b
Xin Liu
659b5f4c2fe7ca485f42285c
Yiyuan Liu
659b5f4c2fe7ca485f42285d
Haoyu Lu
claimed_verified
2025-06-11T08:41:34.930Z
64c206a3fd4d5966b453ed85
https://cdn-avatars.huggingface.co/v1/production/uploads/64c206a3fd4d5966b453ed85/NemBrHcAJFm2ws_VQG8ia.jpeg
Haoyu Lu
user
Nealeon
659b5f4c2fe7ca485f42285e
Shanghao Lu
659b5f4c2fe7ca485f42285f
Fuli Luo
claimed_verified
2024-01-16T08:53:17.789Z
6538815d1bdb3c40db94fbfa
https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/id7aSY8JUgKK2agKWLERt.jpeg
Fuli Luo
user
luofuli
659b5f4c2fe7ca485f422860
Shirong Ma
659b5f4c2fe7ca485f422861
Xiaotao Nie
659b5f4c2fe7ca485f422862
Tian Pei
659b5f4c2fe7ca485f422863
Yishi Piao
659b5f4c2fe7ca485f422864
Junjie Qiu
659b5f4c2fe7ca485f422865
Hui Qu
659b5f4c2fe7ca485f422866
Tongzheng Ren
admin_assigned
2024-01-08T11:34:17.227Z
64f6b67db6365b4e44e5716c
/avatars/f5ef15bfa5c42e3c57f301db7efc38a1.svg
Tongzheng Ren
user
tongzheng
659b5f4c2fe7ca485f422867
Zehui Ren
659b5f4c2fe7ca485f422868
Chong Ruan
claimed_verified
2024-08-17T05:38:34.719Z
6398203609f12714ed1935c2
https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
Chong Ruan
user
Chester111
659b5f4c2fe7ca485f422869
Zhangli Sha
659b5f4c2fe7ca485f42286a
Zhihong Shao
claimed_verified
2024-02-14T08:43:02.469Z
652178662d3ba46ccdf2d5d7
/avatars/bee67db7c23e967becb290f775f8be7d.svg
Zhihong Shao
user
syhia
659b5f4c2fe7ca485f42286b
Junxiao Song
claimed_verified
2024-06-21T07:07:55.381Z
6565a2dd131d13ccc5d8cb12
/avatars/f5c5441ba74791b64c9740911f952bac.svg
Junxiao Song
user
haha-point
659b5f4c2fe7ca485f42286c
Xuecheng Su
659b5f4c2fe7ca485f42286d
Jingxiang Sun
659b5f4c2fe7ca485f42286e
Yaofeng Sun
659b5f4c2fe7ca485f42286f
Minghui Tang
659b5f4c2fe7ca485f422870
Bingxuan Wang
659b5f4c2fe7ca485f422871
Peiyi Wang
659b5f4c2fe7ca485f422872
Shiyu Wang
659b5f4c2fe7ca485f422873
Yaohui Wang
659b5f4c2fe7ca485f422874
Yongji Wang
659b5f4c2fe7ca485f422875
Tong Wu
659b5f4c2fe7ca485f422876
Y. Wu
659b5f4c2fe7ca485f422877
Xin Xie
659b5f4c2fe7ca485f422878
Zhenda Xie
admin_assigned
2024-01-08T11:32:03.475Z
6539f6ea26df26ecd1393c37
https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
Zhenda Xie
user
zdaxie
659b5f4c2fe7ca485f422879
Ziwei Xie
659b5f4c2fe7ca485f42287a
Yiliang Xiong
659b5f4c2fe7ca485f42287b
Hanwei Xu
659b5f4c2fe7ca485f42287c
R. X. Xu
659b5f4c2fe7ca485f42287d
Yanhong Xu
659b5f4c2fe7ca485f42287e
Dejian Yang
admin_assigned
2024-01-08T11:30:51.694Z
6225bb44c6e650de3a65dbaa
/avatars/99c99ced2461978df572c27c1b3a4904.svg
DejianYang
user
DejianYang
659b5f4c2fe7ca485f42287f
Yuxiang You
659b5f4c2fe7ca485f422880
Shuiping Yu
659b5f4c2fe7ca485f422881
Xingkai Yu
659b5f4c2fe7ca485f422882
B. Zhang
659b5f4c2fe7ca485f422883
Haowei Zhang
659b5f4c2fe7ca485f422884
Lecong Zhang
659b5f4c2fe7ca485f422885
Liyue Zhang
659b5f4c2fe7ca485f422886
Mingchuan Zhang
659b5f4c2fe7ca485f422887
Minghua Zhang
659b5f4c2fe7ca485f422888
Wentao Zhang
659b5f4c2fe7ca485f422889
Yichao Zhang
659b5f4c2fe7ca485f42288a
Chenggang Zhao
659b5f4c2fe7ca485f42288b
Yao Zhao
659b5f4c2fe7ca485f42288c
Shangyan Zhou
659b5f4c2fe7ca485f42288d
Shunfeng Zhou
659b5f4c2fe7ca485f42288e
Qihao Zhu
admin_assigned
2024-01-08T11:26:32.148Z
63cd76b4374057a338e8e703
https://cdn-avatars.huggingface.co/v1/production/uploads/63cd76b4374057a338e8e703/i4Qk5-0aYx3oRhC8b50aJ.jpeg
zhuqihao
user
zqh11
659b5f4c2fe7ca485f42288f
Yuheng Zou
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
The rapid development of open-source large language models (LLMs) has been
truly remarkable. However, the scaling law described in previous literature
presents varying conclusions, which casts a dark cloud over scaling LLMs. We
delve into the study of scaling laws and present our distinctive findings that
facilitate scaling of large scale models in two commonly used open-source
configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
LLM, a project dedicated to advancing open-source language models with a
long-term perspective. To support the pre-training phase, we have developed a
dataset that currently consists of 2 trillion tokens and is continuously
expanding. We further conduct supervised fine-tuning (SFT) and Direct
Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
creation of DeepSeek Chat models. Our evaluation results demonstrate that
DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
the domains of code, mathematics, and reasoning. Furthermore, open-ended
evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
compared to GPT-3.5.
DeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.
open-source large language models scaling laws DeepSeek LLM dataset supervised fine-tuning Direct Preference Optimization benchmarks code mathematics reasoning