title: Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language
    Models
summary: Model merging, such as model souping, is the practice of combining different
  models with the same architecture together without further training. In this
  work, we present a model merging methodology that addresses the difficulty of
  fine-tuning Large Language Models (LLMs) for target tasks in non-English
  languages, where task-specific data is often unavailable. We focus on
  mathematical reasoning and without in-language math data, facilitate
  cross-lingual transfer by composing language and math capabilities. Starting
  from the same pretrained model, we fine-tune separate "experts" on math
  instruction data in English and on generic instruction data in the target
  language. We then replace the top and bottom transformer layers of the math
  expert directly with layers from the language expert, which consequently
  enhances math performance in the target language. The resulting merged models
  outperform the individual experts and other merging methods on the math
  benchmark, MGSM, by 10% across four major languages where math instruction data
  is scarce. In addition, this layer swapping is simple, inexpensive, and
  intuitive, as it is based on an interpretative analysis of the most important
  parameter changes during the fine-tuning of each expert. The ability to
  successfully re-compose LLMs for cross-lingual transfer in this manner opens up
  future possibilities to combine model expertise, create modular solutions, and
  transfer reasoning capabilities across languages all post hoc.
paper_authors._id: 6700306bdb3b7a57f9494187
paper_authors.name: Lucas Bandarkar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:02:11.013Z
paper_authors._id: 6435b1d9aaef013d1aeea6a0
paper_authors.avatarUrl: /avatars/1f515c67f3d03f35aa6e06f93566bd21.svg
paper_authors.fullname: Lucas Bandarkar
paper_authors.type: user
paper_authors.user: lucasbandarkar
paper_authors._id: 6700306bdb3b7a57f9494188
paper_authors.name: Benjamin Muller
paper_authors._id: 6700306bdb3b7a57f9494189
paper_authors.name: Pritish Yuvraj
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T20:11:17.248Z
paper_authors._id: 65c30d162a023e7ea97c160f
paper_authors.avatarUrl: /avatars/f17eaa12dd4c4a383fb741f7ae5b9432.svg
paper_authors.fullname: Pritish Yuvraj
paper_authors.type: user
paper_authors.user: pyuvraj
paper_authors._id: 6700306bdb3b7a57f949418a
paper_authors.name: Rui Hou
paper_authors._id: 6700306bdb3b7a57f949418b
paper_authors.name: Nayan Singhal
paper_authors._id: 6700306bdb3b7a57f949418c
paper_authors.name: Hongjiang Lv
paper_authors._id: 6700306bdb3b7a57f949418d
paper_authors.name: Bing Liu
paper_title: Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language
    Models
paper_summary: Model merging, such as model souping, is the practice of combining different
  models with the same architecture together without further training. In this
  work, we present a model merging methodology that addresses the difficulty of
  fine-tuning Large Language Models (LLMs) for target tasks in non-English
  languages, where task-specific data is often unavailable. We focus on
  mathematical reasoning and without in-language math data, facilitate
  cross-lingual transfer by composing language and math capabilities. Starting
  from the same pretrained model, we fine-tune separate "experts" on math
  instruction data in English and on generic instruction data in the target
  language. We then replace the top and bottom transformer layers of the math
  expert directly with layers from the language expert, which consequently
  enhances math performance in the target language. The resulting merged models
  outperform the individual experts and other merging methods on the math
  benchmark, MGSM, by 10% across four major languages where math instruction data
  is scarce. In addition, this layer swapping is simple, inexpensive, and
  intuitive, as it is based on an interpretative analysis of the most important
  parameter changes during the fine-tuning of each expert. The ability to
  successfully re-compose LLMs for cross-lingual transfer in this manner opens up
  future possibilities to combine model expertise, create modular solutions, and
  transfer reasoning capabilities across languages all post hoc.
paper_ai_summary: Merging pretrained models via layer swapping enhances mathematical reasoning in non-English languages where task-specific data is limited.
paper_ai_keywords: model souping Large Language Models (LLMs) mathematical reasoning cross-lingual transfer transformer layers fine-tuning math benchmark MGSM parameter changes model expertise modular solutions