TinyLlama: An Open-Source Small Language Model
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and
tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
open-source community (e.g., FlashAttention), achieving better computational
efficiency. Despite its relatively small size, TinyLlama demonstrates
remarkable performance in a series of downstream tasks. It significantly
outperforms existing open-source language models with comparable sizes. Our
model checkpoints and code are publicly available on GitHub at
https://github.com/jzhang38/TinyLlama.
6597627102a265cc802c0158
Peiyuan Zhang
admin_assigned
2024-01-05T17:20:02.615Z
63565cc56d7fcf1bedb7d347
https://cdn-avatars.huggingface.co/v1/production/uploads/63565cc56d7fcf1bedb7d347/XGcHP4VkO_oieA1gZ4IAX.jpeg
Zhang Peiyuan
user
PY007
6597627102a265cc802c0159
Guangtao Zeng
claimed_verified
2024-01-05T11:31:21.344Z
60ad0de755f970745d4ec28d
/avatars/b0de0222b8ed5fdac8dc7cb0336d2ec7.svg
GtZeng
user
chaoscodes
6597627102a265cc802c015a
Tianduo Wang
claimed_verified
2024-07-29T07:10:19.603Z
6352aa7b6cfb8f149814de5e
https://cdn-avatars.huggingface.co/v1/production/uploads/1666361939036-noauth.jpeg
Tianduo Wang
user
Tianduo
6597627102a265cc802c015b
Wei Lu
TinyLlama: An Open-Source Small Language Model
We present TinyLlama, a compact 1.1B language model pretrained on around 1
trillion tokens for approximately 3 epochs. Building on the architecture and
tokenizer of Llama 2, TinyLlama leverages various advances contributed by the
open-source community (e.g., FlashAttention), achieving better computational
efficiency. Despite its relatively small size, TinyLlama demonstrates
remarkable performance in a series of downstream tasks. It significantly
outperforms existing open-source language models with comparable sizes. Our
model checkpoints and code are publicly available on GitHub at
https://github.com/jzhang38/TinyLlama.
TinyLlama, a compact 1.1B language model, leverages FlashAttention to achieve high performance in downstream tasks with enhanced computational efficiency.
FlashAttention Llama 2 TinyLlama language model