title: TinyLlama: An Open-Source Small Language Model
summary: We present TinyLlama, a compact 1.1B language model pretrained on around 1
  trillion tokens for approximately 3 epochs. Building on the architecture and
  tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
  open-source community (e.g., FlashAttention), achieving better computational
  efficiency. Despite its relatively small size, TinyLlama demonstrates
  remarkable performance in a series of downstream tasks. It significantly
  outperforms existing open-source language models with comparable sizes. Our
  model checkpoints and code are publicly available on GitHub at
  https://github.com/jzhang38/TinyLlama.
paper_authors._id: 6597627102a265cc802c0158
paper_authors.name: Peiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-05T17:20:02.615Z
paper_authors._id: 63565cc56d7fcf1bedb7d347
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
paper_authors.fullname: Zhang Peiyuan
paper_authors.type: user
paper_authors.user: PY007
paper_authors._id: 6597627102a265cc802c0159
paper_authors.name: Guangtao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-05T11:31:21.344Z
paper_authors._id: 60ad0de755f970745d4ec28d
paper_authors.avatarUrl: /avatars/b0de0222b8ed5fdac8dc7cb0336d2ec7.svg
paper_authors.fullname: GtZeng
paper_authors.type: user
paper_authors.user: chaoscodes
paper_authors._id: 6597627102a265cc802c015a
paper_authors.name: Tianduo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-29T07:10:19.603Z
paper_authors._id: 6352aa7b6cfb8f149814de5e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666361939036-noauth.jpeg
paper_authors.fullname: Tianduo Wang
paper_authors.type: user
paper_authors.user: Tianduo
paper_authors._id: 6597627102a265cc802c015b
paper_authors.name: Wei Lu
paper_title: TinyLlama: An Open-Source Small Language Model
paper_summary: We present TinyLlama, a compact 1.1B language model pretrained on around 1
  trillion tokens for approximately 3 epochs. Building on the architecture and
  tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
  open-source community (e.g., FlashAttention), achieving better computational
  efficiency. Despite its relatively small size, TinyLlama demonstrates
  remarkable performance in a series of downstream tasks. It significantly
  outperforms existing open-source language models with comparable sizes. Our
  model checkpoints and code are publicly available on GitHub at
  https://github.com/jzhang38/TinyLlama.
paper_ai_summary: TinyLlama, a compact 1.1B language model, leverages FlashAttention to achieve high performance in downstream tasks with enhanced computational efficiency.
paper_ai_keywords: FlashAttention Llama 2 TinyLlama language model