title: LLM360: Towards Fully Transparent Open-Source LLMs
summary: The recent surge in open-source Large Language Models (LLMs), such as LLaMA,
  Falcon, and Mistral, provides diverse options for AI practitioners and
  researchers. However, most LLMs have only released partial artifacts, such as
  the final model weights or inference code, and technical reports increasingly
  limit their scope to high-level design choices and surface statistics. These
  choices hinder progress in the field by degrading transparency into the
  training of LLMs and forcing teams to rediscover many details in the training
  process. We present LLM360, an initiative to fully open-source LLMs, which
  advocates for all training code and data, model checkpoints, and intermediate
  results to be made available to the community. The goal of LLM360 is to support
  open and collaborative AI research by making the end-to-end LLM training
  process transparent and reproducible by everyone. As a first step of LLM360, we
  release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder,
  including their training code, data, intermediate checkpoints, and analyses (at
  https://www.llm360.ai). We are committed to continually pushing the boundaries
  of LLMs through this open-source effort. More large-scale and stronger models
  are underway and will be released in the future.
paper_authors._id: 6577f192197a6182c338a64f
paper_authors.name: Zhengzhong Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:28:37.331Z
paper_authors._id: 62fbdc67c776fd8821ae3f2d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62fbdc67c776fd8821ae3f2d/cI7iAZOL40RUYluo5ZVTU.png
paper_authors.fullname: Zhengzhong Liu
paper_authors.type: user
paper_authors.user: hunterhector
paper_authors._id: 6577f192197a6182c338a650
paper_authors.name: Aurick Qiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:28:43.735Z
paper_authors._id: 630e8f0bf6f6d700f50ebd2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/630e8f0bf6f6d700f50ebd2e/ELJAinK1XpVJ8aslZ3BIW.jpeg
paper_authors.fullname: Aurick Qiao
paper_authors.type: user
paper_authors.user: aurick
paper_authors._id: 6577f192197a6182c338a651
paper_authors.name: Willie Neiswanger
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-12T08:17:57.765Z
paper_authors._id: 644bf65522d211df6444a7f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
paper_authors.fullname: Willie Neiswanger
paper_authors.type: user
paper_authors.user: willieneis
paper_authors._id: 6577f192197a6182c338a652
paper_authors.name: Hongyi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:29:20.507Z
paper_authors._id: 64e5414fe56d371e42da69c4
paper_authors.avatarUrl: /avatars/7d4a101b0338410bc736e04d7a611aca.svg
paper_authors.fullname: Hongyi Wang
paper_authors.type: user
paper_authors.user: hwang595
paper_authors._id: 6577f192197a6182c338a653
paper_authors.name: Bowen Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:29:35.182Z
paper_authors._id: 64c52182d43e4dee5177e8b7
paper_authors.avatarUrl: /avatars/d13edf5fe3f7f21bb2f22363378b0d9d.svg
paper_authors.fullname: Bowen Tan
paper_authors.type: user
paper_authors.user: btan2
paper_authors._id: 6577f192197a6182c338a654
paper_authors.name: Tianhua Tao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:29:41.752Z
paper_authors._id: 61ecb698382b3b92fea7305f
paper_authors.avatarUrl: /avatars/b1bcf0d9ade12172810cb65dc9c4e4d8.svg
paper_authors.fullname: Tianhua Tao
paper_authors.type: user
paper_authors.user: Tianhua
paper_authors._id: 6577f192197a6182c338a655
paper_authors.name: Junbo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:29:48.130Z
paper_authors._id: 6317d681743def9b3612ebeb
paper_authors.avatarUrl: /avatars/9940b35d5f8ae5df417bb020b609a8d0.svg
paper_authors.fullname: Junbo Li
paper_authors.type: user
paper_authors.user: ljb121002
paper_authors._id: 6577f192197a6182c338a656
paper_authors.name: Yuqi Wang
paper_authors._id: 6577f192197a6182c338a657
paper_authors.name: Suqi Sun
paper_authors._id: 6577f192197a6182c338a658
paper_authors.name: Omkar Pangarkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:30:39.543Z
paper_authors._id: 644086c17663594a126575dd
paper_authors.avatarUrl: /avatars/ba8f2f87bbc3ee7b4da444a4b5097e4c.svg
paper_authors.fullname: Omkar Pangarkar
paper_authors.type: user
paper_authors.user: omkarenator
paper_authors._id: 6577f192197a6182c338a659
paper_authors.name: Richard Fan
paper_authors._id: 6577f192197a6182c338a65a
paper_authors.name: Yi Gu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T07:21:17.481Z
paper_authors._id: 64c2ab94e818eec6128bfeb7
paper_authors.avatarUrl: /avatars/459794e67d43a4b739ddcf90d1921b31.svg
paper_authors.fullname: Yi Gu
paper_authors.type: user
paper_authors.user: wuqing157
paper_authors._id: 6577f192197a6182c338a65b
paper_authors.name: Victor Miller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:31:27.772Z
paper_authors._id: 638a93f1ed88cf97afd53e42
paper_authors.avatarUrl: /avatars/147ed42f13847b5e4d534511ef5388a3.svg
paper_authors.fullname: Victor Miller
paper_authors.type: user
paper_authors.user: victormiller
paper_authors._id: 6577f192197a6182c338a65c
paper_authors.name: Yonghao Zhuang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:31:35.156Z
paper_authors._id: 64ab987073790912c7ad79e6
paper_authors.avatarUrl: /avatars/bfa00418c46da0fd1e522d6561944ec9.svg
paper_authors.fullname: Yonghao Zhuang
paper_authors.type: user
paper_authors.user: ZYHowell
paper_authors._id: 6577f192197a6182c338a65d
paper_authors.name: Guowei He
paper_authors._id: 6577f192197a6182c338a65e
paper_authors.name: Haonan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:32:05.099Z
paper_authors._id: 643934ecb9ac1d55f41cefd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643934ecb9ac1d55f41cefd7/v41FZ2Ar6nVXEkf-w5axf.jpeg
paper_authors.fullname: Haonan Li
paper_authors.type: user
paper_authors.user: haonan-li
paper_authors._id: 6577f192197a6182c338a65f
paper_authors.name: Fajri Koto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:32:15.632Z
paper_authors._id: 643fb246c2ec31af16aa9313
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643fb246c2ec31af16aa9313/zCDYOVGdIrncX-k9o7bM6.png
paper_authors.fullname: Fajri Koto
paper_authors.type: user
paper_authors.user: fajrikoto
paper_authors._id: 6577f192197a6182c338a660
paper_authors.name: Liping Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:32:23.281Z
paper_authors._id: 61a05a0c17ff4c17bba2200f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61a05a0c17ff4c17bba2200f/uMGfYrYubVpmgWXytNHAj.png
paper_authors.fullname: Liping Tang
paper_authors.type: user
paper_authors.user: Liping
paper_authors._id: 6577f192197a6182c338a661
paper_authors.name: Nikhil Ranjan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:32:47.304Z
paper_authors._id: 64de07f2228f31beb858d458
paper_authors.avatarUrl: /avatars/0dfdf90fbd871221466497dc49f6e356.svg
paper_authors.fullname: Nikhil Ranjan
paper_authors.type: user
paper_authors.user: nikhilranjan
paper_authors._id: 6577f192197a6182c338a662
paper_authors.name: Zhiqiang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:32:53.455Z
paper_authors._id: 649d5a3f4e08d59d20014652
paper_authors.avatarUrl: /avatars/017eccd6aec02137c1beebfe4c951720.svg
paper_authors.fullname: Zhiqiang Shen
paper_authors.type: user
paper_authors.user: Jason0214
paper_authors._id: 6577f192197a6182c338a663
paper_authors.name: Xuguang Ren
paper_authors._id: 6577f192197a6182c338a664
paper_authors.name: Roberto Iriondo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:33:13.071Z
paper_authors._id: 652417ebf172523dc3d57eb4
paper_authors.avatarUrl: /avatars/53313181df859b3e7c5eab75211327dd.svg
paper_authors.fullname: Roberto Iriondo
paper_authors.type: user
paper_authors.user: roberto12345
paper_authors._id: 6577f192197a6182c338a665
paper_authors.name: Cun Mu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:33:20.064Z
paper_authors._id: 64f3fc811da608d0842bca3f
paper_authors.avatarUrl: /avatars/b2cd0eae464f577202afd982cd49bd2b.svg
paper_authors.fullname: Cun Mu 
paper_authors.type: user
paper_authors.user: mudongdong
paper_authors._id: 6577f192197a6182c338a666
paper_authors.name: Zhiting Hu
paper_authors._id: 6577f192197a6182c338a667
paper_authors.name: Mark Schulze
paper_authors._id: 6577f192197a6182c338a668
paper_authors.name: Preslav Nakov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:33:39.480Z
paper_authors._id: 647f7eb25e1bc4753746bf9f
paper_authors.avatarUrl: /avatars/cc9c6210fdc822d8a106937e747dff41.svg
paper_authors.fullname: Preslav Nakov
paper_authors.type: user
paper_authors.user: preslavnakov
paper_authors._id: 6577f192197a6182c338a669
paper_authors.name: Tim Baldwin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-23T13:02:28.152Z
paper_authors._id: 65d895d8d435dce78392065f
paper_authors.avatarUrl: /avatars/68c7185d3dd1adf78927b91849978d62.svg
paper_authors.fullname: Tim Baldwin
paper_authors.type: user
paper_authors.user: tbaldwin
paper_authors._id: 6577f192197a6182c338a66a
paper_authors.name: Eric P. Xing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:33:54.472Z
paper_authors._id: 64ff67722ad36636be6c4542
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/sLIrNelAWPVOy4e3oo5LB.jpeg
paper_authors.fullname: Eric Xing
paper_authors.type: user
paper_authors.user: EricX003
paper_title: LLM360: Towards Fully Transparent Open-Source LLMs
paper_summary: The recent surge in open-source Large Language Models (LLMs), such as LLaMA,
  Falcon, and Mistral, provides diverse options for AI practitioners and
  researchers. However, most LLMs have only released partial artifacts, such as
  the final model weights or inference code, and technical reports increasingly
  limit their scope to high-level design choices and surface statistics. These
  choices hinder progress in the field by degrading transparency into the
  training of LLMs and forcing teams to rediscover many details in the training
  process. We present LLM360, an initiative to fully open-source LLMs, which
  advocates for all training code and data, model checkpoints, and intermediate
  results to be made available to the community. The goal of LLM360 is to support
  open and collaborative AI research by making the end-to-end LLM training
  process transparent and reproducible by everyone. As a first step of LLM360, we
  release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder,
  including their training code, data, intermediate checkpoints, and analyses (at
  https://www.llm360.ai). We are committed to continually pushing the boundaries
  of LLMs through this open-source effort. More large-scale and stronger models
  are underway and will be released in the future.
paper_ai_keywords: Large Language Models (LLMs) Open context LLaMA Falcon Mistral Inference code Technical reports High-level design choices Surface statistics Transparency Training of LLMs LLM360 Training code Model checkpoints Intermediate results Open and collaborative AI research End-to-end LLM training process Transparent Reproducible