title: DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
summary: The rapid development of open-source large language models (LLMs) has been
  truly remarkable. However, the scaling law described in previous literature
  presents varying conclusions, which casts a dark cloud over scaling LLMs. We
  delve into the study of scaling laws and present our distinctive findings that
  facilitate scaling of large scale models in two commonly used open-source
  configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
  LLM, a project dedicated to advancing open-source language models with a
  long-term perspective. To support the pre-training phase, we have developed a
  dataset that currently consists of 2 trillion tokens and is continuously
  expanding. We further conduct supervised fine-tuning (SFT) and Direct
  Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
  creation of DeepSeek Chat models. Our evaluation results demonstrate that
  DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
  the domains of code, mathematics, and reasoning. Furthermore, open-ended
  evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
  compared to GPT-3.5.
paper_authors._id: 659b5f4c2fe7ca485f422838
paper_authors.name: DeepSeek-AI
paper_authors._id: 659b5f4c2fe7ca485f42283a
paper_authors.name: Xiao Bi
paper_authors._id: 659b5f4c2fe7ca485f42283b
paper_authors.name: Deli Chen
paper_authors._id: 659b5f4c2fe7ca485f42283c
paper_authors.name: Guanting Chen
paper_authors._id: 659b5f4c2fe7ca485f42283d
paper_authors.name: Shanhuang Chen
paper_authors._id: 659b5f4c2fe7ca485f42283e
paper_authors.name: Damai Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:14:55.147Z
paper_authors._id: 659389f8de82e1ef7b9a8b13
paper_authors.avatarUrl: /avatars/896ed9f4cdbd317493b303d070b7e12a.svg
paper_authors.fullname: Damai Dai
paper_authors.type: user
paper_authors.user: DeepSeekDDM
paper_authors._id: 659b5f4c2fe7ca485f42283f
paper_authors.name: Chengqi Deng
paper_authors._id: 659b5f4c2fe7ca485f422840
paper_authors.name: Honghui Ding
paper_authors._id: 659b5f4c2fe7ca485f422841
paper_authors.name: Kai Dong
paper_authors._id: 659b5f4c2fe7ca485f422842
paper_authors.name: Qiushi Du
paper_authors._id: 659b5f4c2fe7ca485f422843
paper_authors.name: Zhe Fu
paper_authors._id: 659b5f4c2fe7ca485f422844
paper_authors.name: Huazuo Gao
paper_authors._id: 659b5f4c2fe7ca485f422845
paper_authors.name: Kaige Gao
paper_authors._id: 659b5f4c2fe7ca485f422846
paper_authors.name: Wenjun Gao
paper_authors._id: 659b5f4c2fe7ca485f422847
paper_authors.name: Ruiqi Ge
paper_authors._id: 659b5f4c2fe7ca485f422848
paper_authors.name: Kang Guan
paper_authors._id: 659b5f4c2fe7ca485f422849
paper_authors.name: Daya Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:22:02.420Z
paper_authors._id: 653df20eaa1f487614da4db1
paper_authors.avatarUrl: /avatars/12b27ce2c59f53b7e464039deab36a5d.svg
paper_authors.fullname: Daya Guo
paper_authors.type: user
paper_authors.user: guoday
paper_authors._id: 659b5f4c2fe7ca485f42284a
paper_authors.name: Jianzhong Guo
paper_authors._id: 659b5f4c2fe7ca485f42284b
paper_authors.name: Guangbo Hao
paper_authors._id: 659b5f4c2fe7ca485f42284c
paper_authors.name: Zhewen Hao
paper_authors._id: 659b5f4c2fe7ca485f42284d
paper_authors.name: Ying He
paper_authors._id: 659b5f4c2fe7ca485f42284e
paper_authors.name: Wenjie Hu
paper_authors._id: 659b5f4c2fe7ca485f42284f
paper_authors.name: Panpan Huang
paper_authors._id: 659b5f4c2fe7ca485f422850
paper_authors.name: Erhang Li
paper_authors._id: 659b5f4c2fe7ca485f422851
paper_authors.name: Guowei Li
paper_authors._id: 659b5f4c2fe7ca485f422852
paper_authors.name: Jiashi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:23:56.641Z
paper_authors._id: 64fca5f28d50404bc42ca78a
paper_authors.avatarUrl: /avatars/ae01ac0296d6ce1277dacb6894f570b8.svg
paper_authors.fullname: Jiashi Li
paper_authors.type: user
paper_authors.user: Beginlner
paper_authors._id: 659b5f4c2fe7ca485f422853
paper_authors.name: Yao Li
paper_authors._id: 659b5f4c2fe7ca485f422854
paper_authors.name: Y. K. Li
paper_authors._id: 659b5f4c2fe7ca485f422855
paper_authors.name: Wenfeng Liang
paper_authors._id: 659b5f4c2fe7ca485f422856
paper_authors.name: Fangyun Lin
paper_authors._id: 659b5f4c2fe7ca485f422857
paper_authors.name: A. X. Liu
paper_authors._id: 659b5f4c2fe7ca485f422858
paper_authors.name: Bo Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-08T21:57:26.682Z
paper_authors._id: 635e3a76106f984574c36409
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667120725800-635e3a76106f984574c36409.png
paper_authors.fullname: Bo Liu
paper_authors.type: user
paper_authors.user: Benjamin-eecs
paper_authors._id: 659b5f4c2fe7ca485f422859
paper_authors.name: Wen Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-22T10:31:39.208Z
paper_authors._id: 63198e8802fb322037332f2d
paper_authors.avatarUrl: /avatars/d3f9c206e387df35beb0ed0ef1cdf865.svg
paper_authors.fullname: Wen Liu
paper_authors.type: user
paper_authors.user: doubility123
paper_authors._id: 659b5f4c2fe7ca485f42285a
paper_authors.name: Xiaodong Liu
paper_authors._id: 659b5f4c2fe7ca485f42285b
paper_authors.name: Xin Liu
paper_authors._id: 659b5f4c2fe7ca485f42285c
paper_authors.name: Yiyuan Liu
paper_authors._id: 659b5f4c2fe7ca485f42285d
paper_authors.name: Haoyu Lu
paper_authors._id: 659b5f4c2fe7ca485f42285e
paper_authors.name: Shanghao Lu
paper_authors._id: 659b5f4c2fe7ca485f42285f
paper_authors.name: Fuli Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-16T08:53:17.789Z
paper_authors._id: 6538815d1bdb3c40db94fbfa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6538815d1bdb3c40db94fbfa/id7aSY8JUgKK2agKWLERt.jpeg
paper_authors.fullname: Fuli Luo
paper_authors.type: user
paper_authors.user: luofuli
paper_authors._id: 659b5f4c2fe7ca485f422860
paper_authors.name: Shirong Ma
paper_authors._id: 659b5f4c2fe7ca485f422861
paper_authors.name: Xiaotao Nie
paper_authors._id: 659b5f4c2fe7ca485f422862
paper_authors.name: Tian Pei
paper_authors._id: 659b5f4c2fe7ca485f422863
paper_authors.name: Yishi Piao
paper_authors._id: 659b5f4c2fe7ca485f422864
paper_authors.name: Junjie Qiu
paper_authors._id: 659b5f4c2fe7ca485f422865
paper_authors.name: Hui Qu
paper_authors._id: 659b5f4c2fe7ca485f422866
paper_authors.name: Tongzheng Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:34:17.227Z
paper_authors._id: 64f6b67db6365b4e44e5716c
paper_authors.avatarUrl: /avatars/f5ef15bfa5c42e3c57f301db7efc38a1.svg
paper_authors.fullname: Tongzheng Ren
paper_authors.type: user
paper_authors.user: tongzheng
paper_authors._id: 659b5f4c2fe7ca485f422867
paper_authors.name: Zehui Ren
paper_authors._id: 659b5f4c2fe7ca485f422868
paper_authors.name: Chong Ruan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-17T05:38:34.719Z
paper_authors._id: 6398203609f12714ed1935c2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6398203609f12714ed1935c2/uXgl0LgKnFYjq1Wz39-a6.jpeg
paper_authors.fullname: Chong Ruan
paper_authors.type: user
paper_authors.user: Chester111
paper_authors._id: 659b5f4c2fe7ca485f422869
paper_authors.name: Zhangli Sha
paper_authors._id: 659b5f4c2fe7ca485f42286a
paper_authors.name: Zhihong Shao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-14T08:43:02.469Z
paper_authors._id: 652178662d3ba46ccdf2d5d7
paper_authors.avatarUrl: /avatars/bee67db7c23e967becb290f775f8be7d.svg
paper_authors.fullname: Zhihong Shao
paper_authors.type: user
paper_authors.user: syhia
paper_authors._id: 659b5f4c2fe7ca485f42286b
paper_authors.name: Junxiao Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:55.381Z
paper_authors._id: 6565a2dd131d13ccc5d8cb12
paper_authors.avatarUrl: /avatars/f5c5441ba74791b64c9740911f952bac.svg
paper_authors.fullname: Junxiao Song
paper_authors.type: user
paper_authors.user: haha-point
paper_authors._id: 659b5f4c2fe7ca485f42286c
paper_authors.name: Xuecheng Su
paper_authors._id: 659b5f4c2fe7ca485f42286d
paper_authors.name: Jingxiang Sun
paper_authors._id: 659b5f4c2fe7ca485f42286e
paper_authors.name: Yaofeng Sun
paper_authors._id: 659b5f4c2fe7ca485f42286f
paper_authors.name: Minghui Tang
paper_authors._id: 659b5f4c2fe7ca485f422870
paper_authors.name: Bingxuan Wang
paper_authors._id: 659b5f4c2fe7ca485f422871
paper_authors.name: Peiyi Wang
paper_authors._id: 659b5f4c2fe7ca485f422872
paper_authors.name: Shiyu Wang
paper_authors._id: 659b5f4c2fe7ca485f422873
paper_authors.name: Yaohui Wang
paper_authors._id: 659b5f4c2fe7ca485f422874
paper_authors.name: Yongji Wang
paper_authors._id: 659b5f4c2fe7ca485f422875
paper_authors.name: Tong Wu
paper_authors._id: 659b5f4c2fe7ca485f422876
paper_authors.name: Y. Wu
paper_authors._id: 659b5f4c2fe7ca485f422877
paper_authors.name: Xin Xie
paper_authors._id: 659b5f4c2fe7ca485f422878
paper_authors.name: Zhenda Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:32:03.475Z
paper_authors._id: 6539f6ea26df26ecd1393c37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6539f6ea26df26ecd1393c37/9VJusLLAiLhUxfBAFggpF.jpeg
paper_authors.fullname: Zhenda Xie
paper_authors.type: user
paper_authors.user: zdaxie
paper_authors._id: 659b5f4c2fe7ca485f422879
paper_authors.name: Ziwei Xie
paper_authors._id: 659b5f4c2fe7ca485f42287a
paper_authors.name: Yiliang Xiong
paper_authors._id: 659b5f4c2fe7ca485f42287b
paper_authors.name: Hanwei Xu
paper_authors._id: 659b5f4c2fe7ca485f42287c
paper_authors.name: R. X. Xu
paper_authors._id: 659b5f4c2fe7ca485f42287d
paper_authors.name: Yanhong Xu
paper_authors._id: 659b5f4c2fe7ca485f42287e
paper_authors.name: Dejian Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:30:51.694Z
paper_authors._id: 6225bb44c6e650de3a65dbaa
paper_authors.avatarUrl: /avatars/99c99ced2461978df572c27c1b3a4904.svg
paper_authors.fullname: DejianYang
paper_authors.type: user
paper_authors.user: DejianYang
paper_authors._id: 659b5f4c2fe7ca485f42287f
paper_authors.name: Yuxiang You
paper_authors._id: 659b5f4c2fe7ca485f422880
paper_authors.name: Shuiping Yu
paper_authors._id: 659b5f4c2fe7ca485f422881
paper_authors.name: Xingkai Yu
paper_authors._id: 659b5f4c2fe7ca485f422882
paper_authors.name: B. Zhang
paper_authors._id: 659b5f4c2fe7ca485f422883
paper_authors.name: Haowei Zhang
paper_authors._id: 659b5f4c2fe7ca485f422884
paper_authors.name: Lecong Zhang
paper_authors._id: 659b5f4c2fe7ca485f422885
paper_authors.name: Liyue Zhang
paper_authors._id: 659b5f4c2fe7ca485f422886
paper_authors.name: Mingchuan Zhang
paper_authors._id: 659b5f4c2fe7ca485f422887
paper_authors.name: Minghua Zhang
paper_authors._id: 659b5f4c2fe7ca485f422888
paper_authors.name: Wentao Zhang
paper_authors._id: 659b5f4c2fe7ca485f422889
paper_authors.name: Yichao Zhang
paper_authors._id: 659b5f4c2fe7ca485f42288a
paper_authors.name: Chenggang Zhao
paper_authors._id: 659b5f4c2fe7ca485f42288b
paper_authors.name: Yao Zhao
paper_authors._id: 659b5f4c2fe7ca485f42288c
paper_authors.name: Shangyan Zhou
paper_authors._id: 659b5f4c2fe7ca485f42288d
paper_authors.name: Shunfeng Zhou
paper_authors._id: 659b5f4c2fe7ca485f42288e
paper_authors.name: Qihao Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-08T11:26:32.148Z
paper_authors._id: 63cd76b4374057a338e8e703
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63cd76b4374057a338e8e703/i4Qk5-0aYx3oRhC8b50aJ.jpeg
paper_authors.fullname: zhuqihao
paper_authors.type: user
paper_authors.user: zqh11
paper_authors._id: 659b5f4c2fe7ca485f42288f
paper_authors.name: Yuheng Zou
paper_title: DeepSeek LLM: Scaling Open-Source Language Models with Longtermism
paper_summary: The rapid development of open-source large language models (LLMs) has been
  truly remarkable. However, the scaling law described in previous literature
  presents varying conclusions, which casts a dark cloud over scaling LLMs. We
  delve into the study of scaling laws and present our distinctive findings that
  facilitate scaling of large scale models in two commonly used open-source
  configurations, 7B and 67B. Guided by the scaling laws, we introduce DeepSeek
  LLM, a project dedicated to advancing open-source language models with a
  long-term perspective. To support the pre-training phase, we have developed a
  dataset that currently consists of 2 trillion tokens and is continuously
  expanding. We further conduct supervised fine-tuning (SFT) and Direct
  Preference Optimization (DPO) on DeepSeek LLM Base models, resulting in the
  creation of DeepSeek Chat models. Our evaluation results demonstrate that
  DeepSeek LLM 67B surpasses LLaMA-2 70B on various benchmarks, particularly in
  the domains of code, mathematics, and reasoning. Furthermore, open-ended
  evaluations reveal that DeepSeek LLM 67B Chat exhibits superior performance
  compared to GPT-3.5.
paper_ai_summary: DeepSeek LLM, an open-source language model project, develops a large dataset and employs SFT and DPO to achieve performance surpassing LLaMA-2 70B and GPT-3.5 in various benchmarks and open-ended evaluations.
paper_ai_keywords: open-source large language models scaling laws DeepSeek LLM dataset supervised fine-tuning Direct Preference Optimization benchmarks code mathematics reasoning