title: BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
summary: We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new
  state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was
  trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and
  8,192 context lengths. BTLM-3B-8K outperforms all existing 3B parameter models
  by 2-5.5% across downstream tasks. BTLM-3B-8K is even competitive with some 7B
  parameter models. Additionally, BTLM-3B-8K provides excellent long context
  performance, outperforming MPT-7B-8K and XGen-7B-8K on tasks up to 8,192
  context length. We trained the model on a cleaned and deduplicated SlimPajama
  dataset; aggressively tuned the \textmu P hyperparameters and schedule; used
  ALiBi position embeddings; and adopted the SwiGLU nonlinearity.
    On Hugging Face, the most popular models have 7B parameters, indicating that
  users prefer the quality-size ratio of 7B models. Compacting the 7B parameter
  model to one with 3B parameters, with little performance impact, is an
  important milestone. BTLM-3B-8K needs only 3GB of memory with 4-bit precision
  and takes 2.5x less inference compute than 7B models, helping to open up access
  to a powerful language model on mobile and edge devices. BTLM-3B-8K is
  available under an Apache 2.0 license on Hugging Face:
  https://huggingface.co/cerebras/btlm-3b-8k-base.
paper_authors._id: 650d015be2c7c01fc1a62765
paper_authors.name: Nolan Dey
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:29:02.892Z
paper_authors._id: 62c47a267877fdc687c613da
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62c47a267877fdc687c613da/G0uoHHgEuND3Ro5zU1Kk5.jpeg
paper_authors.fullname: Nolan Dey
paper_authors.type: user
paper_authors.user: ndey96
paper_authors._id: 650d015be2c7c01fc1a62766
paper_authors.name: Daria Soboleva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:29:50.231Z
paper_authors._id: 6414bbe095fb6f824b2035a5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679080567124-6414bbe095fb6f824b2035a5.jpeg
paper_authors.fullname: Daria Soboleva
paper_authors.type: user
paper_authors.user: daria-soboleva
paper_authors._id: 650d015be2c7c01fc1a62767
paper_authors.name: Faisal Al-Khateeb
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:29:57.959Z
paper_authors._id: 6419da12b4adb0e101b22ac3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6419da12b4adb0e101b22ac3/mI4FjJCcgv4vDQvmXlFrm.jpeg
paper_authors.fullname: Faisal Al-khateeb
paper_authors.type: user
paper_authors.user: faisalak
paper_authors._id: 650d015be2c7c01fc1a62768
paper_authors.name: Bowen Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:30:20.855Z
paper_authors._id: 64c4e59619565937fb279a94
paper_authors.avatarUrl: /avatars/201c31460110d4d8e5f5896ff920e382.svg
paper_authors.fullname: Bowen Yang
paper_authors.type: user
paper_authors.user: Bowenyang-online
paper_authors._id: 650d015be2c7c01fc1a62769
paper_authors.name: Ribhu Pathria
paper_authors._id: 650d015be2c7c01fc1a6276a
paper_authors.name: Hemant Khachane
paper_authors._id: 650d015be2c7c01fc1a6276b
paper_authors.name: Shaheer Muhammad
paper_authors._id: 650d015be2c7c01fc1a6276c
paper_authors.name: Zhiming
paper_authors._id: 650d015be2c7c01fc1a6276d
paper_authors.name: Chen
paper_authors._id: 650d015be2c7c01fc1a6276e
paper_authors.name: Robert Myers
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:31:44.102Z
paper_authors._id: 6132c6c146a28b5058423152
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6132c6c146a28b5058423152/uCf-HNZnG3CNo9mNwzgQ-.jpeg
paper_authors.fullname: Rob Myers
paper_authors.type: user
paper_authors.user: robertmyers
paper_authors._id: 650d015be2c7c01fc1a6276f
paper_authors.name: Jacob Robert Steeves
paper_authors._id: 650d015be2c7c01fc1a62770
paper_authors.name: Natalia Vassilieva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:32:16.718Z
paper_authors._id: 6410072c208821a59b780fcb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6410072c208821a59b780fcb/R9ZDtB9KIaNrLmM2b0GKc.png
paper_authors.fullname: Natalia Vassilieva
paper_authors.type: user
paper_authors.user: vnata
paper_authors._id: 650d015be2c7c01fc1a62771
paper_authors.name: Marvin Tom
paper_authors._id: 650d015be2c7c01fc1a62772
paper_authors.name: Joel Hestness
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-22T15:32:34.195Z
paper_authors._id: 643433124b34368fdb02e8b0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643433124b34368fdb02e8b0/mnBzw2JSHk0I5vXGY-YVI.jpeg
paper_authors.fullname: Joel Hestness
paper_authors.type: user
paper_authors.user: jthestness
paper_title: BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
paper_summary: We introduce the Bittensor Language Model, called "BTLM-3B-8K", a new
  state-of-the-art 3 billion parameter open-source language model. BTLM-3B-8K was
  trained on 627B tokens from the SlimPajama dataset with a mixture of 2,048 and
  8,192 context lengths. BTLM-3B-8K outperforms all existing 3B parameter models
  by 2-5.5% across downstream tasks. BTLM-3B-8K is even competitive with some 7B
  parameter models. Additionally, BTLM-3B-8K provides excellent long context
  performance, outperforming MPT-7B-8K and XGen-7B-8K on tasks up to 8,192
  context length. We trained the model on a cleaned and deduplicated SlimPajama
  dataset; aggressively tuned the \textmu P hyperparameters and schedule; used
  ALiBi position embeddings; and adopted the SwiGLU nonlinearity.
    On Hugging Face, the most popular models have 7B parameters, indicating that
  users prefer the quality-size ratio of 7B models. Compacting the 7B parameter
  model to one with 3B parameters, with little performance impact, is an
  important milestone. BTLM-3B-8K needs only 3GB of memory with 4-bit precision
  and takes 2.5x less inference compute than 7B models, helping to open up access
  to a powerful language model on mobile and edge devices. BTLM-3B-8K is
  available under an Apache 2.0 license on Hugging Face:
  https://huggingface.co/cerebras/btlm-3b-8k-base.
paper_ai_keywords: deep learning language model parameter-efficient fine-tuning SlimPajama dataset context length long context performance MPT-7B-8K XGen-7B-8K position embeddings ALiBi position embeddings SwiGLU nonlinearity hyperparameters schedule parameter-efficient fine-tuning memory efficiency inference compute mobile devices edge devices Apache 2.0 license