Trillion 7B Technical Report
We introduce Trillion-7B, the most token-efficient Korean-centric
multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA)
mechanism enables highly efficient and effective knowledge transfer from
English to target languages like Korean and Japanese. Combined with optimized
data mixtures, language-specific filtering, and tailored tokenizer
construction, Trillion-7B achieves competitive performance while dedicating
only 10\% of its 2T training tokens to multilingual data and requiring just
59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations
across 27 benchmarks in four languages demonstrate Trillion-7B's robust
multilingual performance and exceptional cross-lingual consistency.
680879ead6dc8bf64565c975
Sungjun Han
admin_assigned
2025-04-24T11:23:37.021Z
67aaee60a8192c1ba3d7d42b
https://cdn-avatars.huggingface.co/v1/production/uploads/67aaee60a8192c1ba3d7d42b/2pBbWeqUEUyB82EPJpNX4.jpeg
Sungjun Han
user
sungjunhan-trl
680879ead6dc8bf64565c976
Juyoung Suk
claimed_verified
2025-04-23T08:08:21.257Z
6138cc1306dd10833d2db64b
https://cdn-avatars.huggingface.co/v1/production/uploads/6138cc1306dd10833d2db64b/IRX4y-8M4YlzR_8jOwkKp.jpeg
Juyoung Suk
user
juyoungml
680879ead6dc8bf64565c977
Suyeong An
680879ead6dc8bf64565c978
Hyungguk Kim
680879ead6dc8bf64565c979
Kyuseok Kim
admin_assigned
2025-04-24T11:24:00.523Z
6729c4baae4312f3ab001a90
/avatars/5d454f1695a0a8574e730fd4d884243b.svg
Kyuseok Kim
user
kyudolski
680879ead6dc8bf64565c97a
Wonsuk Yang
680879ead6dc8bf64565c97b
Seungtaek Choi
claimed_verified
2025-04-24T09:11:03.864Z
6257adfdb98dcaa7e0de7ab4
/avatars/ddfc2135104895d09cfce0cd6f10e5fb.svg
Seungtaek Choi
user
hist0613
680879ead6dc8bf64565c97c
Jamin Shin
claimed_verified
2025-04-24T12:41:14.275Z
6188cf3293317afcd1c2df7f
/avatars/95621801d5b3f3c1a681f1ad6cc66c6a.svg
Jay Shin
user
jshin49
Trillion 7B Technical Report
We introduce Trillion-7B, the most token-efficient Korean-centric
multilingual LLM available. Our novel Cross-lingual Document Attention (XLDA)
mechanism enables highly efficient and effective knowledge transfer from
English to target languages like Korean and Japanese. Combined with optimized
data mixtures, language-specific filtering, and tailored tokenizer
construction, Trillion-7B achieves competitive performance while dedicating
only 10\% of its 2T training tokens to multilingual data and requiring just
59.4K H100 GPU hours (\$148K) for full training. Comprehensive evaluations
across 27 benchmarks in four languages demonstrate Trillion-7B's robust
multilingual performance and exceptional cross-lingual consistency.
Trillion-7B is a highly efficient multilingual LLM leveraging Cross-lingual Document Attention (XLDA) for knowledge transfer and achieving competitive performance with minimal multilingual training data.
Cross-lingual Document Attention (XLDA)