title: PolyLM: An Open Source Polyglot Large Language Model
summary: Large language models (LLMs) demonstrate remarkable ability to comprehend,
  reason, and generate following nature language instructions. However, the
  development of LLMs has been primarily focused on high-resource languages, such
  as English, thereby limiting their applicability and research in other
  languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
  billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
  multilingual capabilities, we 1) integrate bilingual data into training data;
  and 2) adopt a curriculum learning strategy that increases the proportion of
  non-English data from 30% in the first stage to 60% in the final stage during
  pre-training. Further, we propose a multilingual self-instruct method which
  automatically generates 132.7K diverse multilingual instructions for model
  fine-tuning. To assess the model's performance, we collect several existing
  multilingual tasks, including multilingual understanding, question answering,
  generation, and translation. Extensive experiments show that PolyLM surpasses
  other open-source models such as LLaMA and BLOOM on multilingual tasks while
  maintaining comparable performance in English. Our models, alone with the
  instruction data and multilingual benchmark, are available at:
  https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.
paper_authors._id: 64af6223ddf1f172eb2d6770
paper_authors.name: Xiangpeng Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T08:32:03.355Z
paper_authors._id: 64abad4d329f5e6e7831ced4
paper_authors.avatarUrl: /avatars/7702da77650df0be9f21cc300f761975.svg
paper_authors.fullname: Xiangpeng Wei
paper_authors.type: user
paper_authors.user: pemywei
paper_authors._id: 64af6223ddf1f172eb2d6771
paper_authors.name: Haoran Wei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:14:01.496Z
paper_authors._id: 628b430afc0078a72e38b04a
paper_authors.avatarUrl: /avatars/b4958c184be06534645f2284635d850e.svg
paper_authors.fullname: Haoran Wei
paper_authors.type: user
paper_authors.user: whr94621
paper_authors._id: 64af6223ddf1f172eb2d6772
paper_authors.name: Huan Lin
paper_authors._id: 64af6223ddf1f172eb2d6773
paper_authors.name: Tianhao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:16:19.015Z
paper_authors._id: 64abb96bd691b1c2482e7c19
paper_authors.avatarUrl: /avatars/ff3a429f985a52c9c5ea4f64872599f2.svg
paper_authors.fullname: litianhao
paper_authors.type: user
paper_authors.user: litianhao
paper_authors._id: 64af6223ddf1f172eb2d6774
paper_authors.name: Pei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T10:16:49.434Z
paper_authors._id: 641190aebeea38a6513bc558
paper_authors.avatarUrl: /avatars/ff2f97adf98861de563017d3589961f1.svg
paper_authors.fullname: Pei Zhang
paper_authors.type: user
paper_authors.user: psyangqi
paper_authors._id: 64af6223ddf1f172eb2d6775
paper_authors.name: Xingzhang Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-03T14:07:15.564Z
paper_authors._id: 645c82c4081be4b32049633a
paper_authors.avatarUrl: /avatars/e5a08cf0ec5a04bd9d66111382ce0508.svg
paper_authors.fullname: xzhren
paper_authors.type: user
paper_authors.user: xingzhang
paper_authors._id: 64af6223ddf1f172eb2d6776
paper_authors.name: Mei Li
paper_authors._id: 64af6223ddf1f172eb2d6777
paper_authors.name: Yu Wan
paper_authors._id: 64af6223ddf1f172eb2d6778
paper_authors.name: Zhiwei Cao
paper_authors._id: 64af6223ddf1f172eb2d6779
paper_authors.name: Binbin Xie
paper_authors._id: 64af6223ddf1f172eb2d677a
paper_authors.name: Tianxiang Hu
paper_authors._id: 64af6223ddf1f172eb2d677b
paper_authors.name: Shangjie Li
paper_authors._id: 64af6223ddf1f172eb2d677c
paper_authors.name: Binyuan Hui
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T03:33:47.001Z
paper_authors._id: 61e4c4ca1ab24785ac11ba69
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61e4c4ca1ab24785ac11ba69/1Q1zhhyGSJ9RJG9MzwxVv.jpeg
paper_authors.fullname: Binyuan Hui
paper_authors.type: user
paper_authors.user: huybery
paper_authors._id: 64af6223ddf1f172eb2d677d
paper_authors.name: Bowen Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:47.381Z
paper_authors._id: 63d9d68c1cae35c27bf7a6a7
paper_authors.avatarUrl: /avatars/b5ad98cf269ae5f1fe90861fb4170fae.svg
paper_authors.fullname: Bowen Yu
paper_authors.type: user
paper_authors.user: Tigerph
paper_authors._id: 64af6223ddf1f172eb2d677e
paper_authors.name: Dayiheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-20T08:34:52.376Z
paper_authors._id: 6434d4989bd5a84b5dd0b0f5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6434d4989bd5a84b5dd0b0f5/0Elf9qbfG9Hkgypm9pTGm.jpeg
paper_authors.fullname: Dayiheng Liu
paper_authors.type: user
paper_authors.user: Losin94
paper_authors._id: 64af6223ddf1f172eb2d677f
paper_authors.name: Baosong Yang
paper_authors._id: 64af6223ddf1f172eb2d6780
paper_authors.name: Fei Huang
paper_authors._id: 64af6223ddf1f172eb2d6781
paper_authors.name: Jun Xie
paper_title: PolyLM: An Open Source Polyglot Large Language Model
paper_summary: Large language models (LLMs) demonstrate remarkable ability to comprehend,
  reason, and generate following nature language instructions. However, the
  development of LLMs has been primarily focused on high-resource languages, such
  as English, thereby limiting their applicability and research in other
  languages. Consequently, we present PolyLM, a multilingual LLM trained on 640
  billion (B) tokens, avaliable in two model sizes: 1.7B and 13B. To enhance its
  multilingual capabilities, we 1) integrate bilingual data into training data;
  and 2) adopt a curriculum learning strategy that increases the proportion of
  non-English data from 30% in the first stage to 60% in the final stage during
  pre-training. Further, we propose a multilingual self-instruct method which
  automatically generates 132.7K diverse multilingual instructions for model
  fine-tuning. To assess the model's performance, we collect several existing
  multilingual tasks, including multilingual understanding, question answering,
  generation, and translation. Extensive experiments show that PolyLM surpasses
  other open-source models such as LLaMA and BLOOM on multilingual tasks while
  maintaining comparable performance in English. Our models, alone with the
  instruction data and multilingual benchmark, are available at:
  https://modelscope.cn/models/damo/nlp_polylm_13b_text_generation.
paper_ai_keywords: Large language models (LLMs) multilingual LLM pretrained curriculum learning multilingual self-instruct multilingual understanding multilingual question answering multilingual generation multilingual translation open-source models (LLaMA and BLOOM) multilingual benchmark