Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models
<|reference_start|>Extracting and Transferring Abilities For Building Multi-lingual Ability-enhanced Large Language Models: Multi-lingual ability transfer has become increasingly important for the broad application of large language models (LLMs). Existing work highly relies on training with the multi-lingual ability-related data, which may be not available for low-resource languages. To solve it, we propose a Multi-lingual Ability Extraction and Transfer approach, named as MAET. Our key idea is to decompose and extract language-agnostic ability-related weights from LLMs, and transfer them across different languages by simple addition and subtraction operations without training. Specially, our MAET consists of the extraction and transfer stages. In the extraction stage, we firstly locate key neurons that are highly related to specific abilities, and then employ them to extract the transferable ability-specific weights. In the transfer stage, we further select the ability-related parameter tensors, and design the merging strategy based on the linguistic and ability specific weights, to build the multi-lingual ability-enhanced LLM. To demonstrate the effectiveness of our proposed approach, we conduct extensive experiments on mathematical and scientific tasks in both high-resource lingual and low-resource lingual scenarios. Experiment results have shown that MAET can effectively and efficiently extract and transfer the advanced abilities, and outperform training-based baseline methods. Our code and data are available at \url{https://github.com/RUCAIBox/MAET}.<|reference_end|>
@article{chen2024extracting,
    title={Extracting and Transferring Abilities For Building Multi-lingual
  Ability-enhanced Large Language Models},
    author={Zhipeng Chen, Liang Song, Kun Zhou, Wayne Xin Zhao, Bingning Wang,
  Weipeng Chen, Ji-Rong Wen},
    journal={arXiv preprint arXiv:2410.07825},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.07825},
    primaryClass={cs.CL}
}