PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency
<|reference_start|>PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency: We introduce PLaMo-100B, a large-scale language model designed for Japanese proficiency. The model was trained from scratch using 2 trillion tokens, with architecture such as QK Normalization and Z-Loss to ensure training stability during the training process. Post-training techniques, including Supervised Fine-Tuning and Direct Preference Optimization, were applied to refine the model's performance. Benchmark evaluations suggest that PLaMo-100B performs well, particularly in Japanese-specific tasks, achieving results that are competitive with frontier models like GPT-4.<|reference_end|>
@article{elements2024plamo-100b:,
    title={PLaMo-100B: A Ground-Up Language Model Designed for Japanese Proficiency},
    author={Preferred Elements: Kenshin Abe, Kaizaburo Chubachi, Yasuhiro Fujita,
  Yuta Hirokawa, Kentaro Imajo, Toshiki Kataoka, Hiroyoshi Komatsu, Hiroaki
  Mikami, Tsuguo Mogami, Shogo Murai, Kosuke Nakago, Daisuke Nishino, Toru
  Ogawa, Daisuke Okanohara, Yoshihiko Ozaki, Shotaro Sano, Shuji Suzuki, Tianqi
  Xu, Toshihiko Yanase},
    journal={arXiv preprint arXiv:2410.07563},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07563},
    primaryClass={cs.CL cs.AI cs.LG}
}