title: FuxiTranyu: A Multilingual Large Language Model Trained with Balanced
    Data
summary: Large language models (LLMs) have demonstrated prowess in a wide range of
  tasks. However, many LLMs exhibit significant performance discrepancies between
  high- and low-resource languages. To mitigate this challenge, we present
  FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the
  need of the research community for balanced and high-performing multilingual
  capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is
  trained from scratch on a meticulously balanced multilingual data repository
  that contains 600 billion tokens covering 43 natural languages and 16
  programming languages. In addition to the base model, we also develop two
  instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse
  multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined
  with DPO on a preference dataset for enhanced alignment ability. Extensive
  experiments on a wide range of multilingual benchmarks demonstrate the
  competitive performance of FuxiTranyu against existing multilingual LLMs, e.g.,
  BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability
  analyses at both the neuron and representation level suggest that FuxiTranyu is
  able to learn consistent multilingual representations across different
  languages. To promote further research into multilingual LLMs and their working
  mechanisms, we release both the base and instruction-tuned FuxiTranyu models
  together with 58 pretraining checkpoints at HuggingFace and Github.
paper_authors._id: 66bc35369d958666bc9bf156
paper_authors.name: Haoran Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-14T13:03:39.825Z
paper_authors._id: 669c82a44bbe8ad52ea113a5
paper_authors.avatarUrl: /avatars/809256e0586e05fa754047322d3ccc68.svg
paper_authors.fullname: Haoran Sun
paper_authors.type: user
paper_authors.user: hrsun15
paper_authors._id: 66bc35369d958666bc9bf157
paper_authors.name: Renren Jin
paper_authors._id: 66bc35369d958666bc9bf158
paper_authors.name: Shaoyang Xu
paper_authors._id: 66bc35369d958666bc9bf159
paper_authors.name: Leiyu Pan
paper_authors._id: 66bc35369d958666bc9bf15a
paper_authors.name: Supryadi
paper_authors._id: 66bc35369d958666bc9bf15b
paper_authors.name: Menglong Cui
paper_authors._id: 66bc35369d958666bc9bf15c
paper_authors.name: Jiangcun Du
paper_authors._id: 66bc35369d958666bc9bf15d
paper_authors.name: Yikun Lei
paper_authors._id: 66bc35369d958666bc9bf15e
paper_authors.name: Lei Yang
paper_authors._id: 66bc35369d958666bc9bf15f
paper_authors.name: Ling Shi
paper_authors._id: 66bc35369d958666bc9bf160
paper_authors.name: Juesi Xiao
paper_authors._id: 66bc35369d958666bc9bf161
paper_authors.name: Shaolin Zhu
paper_authors._id: 66bc35369d958666bc9bf162
paper_authors.name: Deyi Xiong
paper_title: FuxiTranyu: A Multilingual Large Language Model Trained with Balanced
    Data
paper_summary: Large language models (LLMs) have demonstrated prowess in a wide range of
  tasks. However, many LLMs exhibit significant performance discrepancies between
  high- and low-resource languages. To mitigate this challenge, we present
  FuxiTranyu, an open-source multilingual LLM, which is designed to satisfy the
  need of the research community for balanced and high-performing multilingual
  capabilities. FuxiTranyu-8B, the base model with 8 billion parameters, is
  trained from scratch on a meticulously balanced multilingual data repository
  that contains 600 billion tokens covering 43 natural languages and 16
  programming languages. In addition to the base model, we also develop two
  instruction-tuned models: FuxiTranyu-8B-SFT that is fine-tuned on a diverse
  multilingual instruction dataset, and FuxiTranyu-8B-DPO that is further refined
  with DPO on a preference dataset for enhanced alignment ability. Extensive
  experiments on a wide range of multilingual benchmarks demonstrate the
  competitive performance of FuxiTranyu against existing multilingual LLMs, e.g.,
  BLOOM-7B, PolyLM-13B, Llama-2-Chat-7B and Mistral-7B-Instruct. Interpretability
  analyses at both the neuron and representation level suggest that FuxiTranyu is
  able to learn consistent multilingual representations across different
  languages. To promote further research into multilingual LLMs and their working
  mechanisms, we release both the base and instruction-tuned FuxiTranyu models
  together with 58 pretraining checkpoints at HuggingFace and Github.
paper_ai_keywords: multilingual LLM fine-tuning instruction-tuned DPO (Direct Preference Optimization) multilingual benchmarks multilingual representations neuron-level interpretability representation-level interpretability base model pretraining checkpoints