MindMerger: Efficient Boosting LLM Reasoning in non-English Languages
<|reference_start|>MindMerger: Efficient Boosting LLM Reasoning in non-English Languages: Reasoning capabilities are crucial for Large Language Models (LLMs), yet a notable gap exists between English and non-English languages. To bridge this disparity, some works fine-tune LLMs to relearn reasoning capabilities in non-English languages, while others replace non-English inputs with an external model's outputs such as English translation text to circumvent the challenge of LLM understanding non-English. Unfortunately, these methods often underutilize the built-in skilled reasoning and useful language understanding capabilities of LLMs. In order to better utilize the minds of reasoning and language understanding in LLMs, we propose a new method, namely MindMerger, which merges LLMs with the external language understanding capabilities from multilingual models to boost the multilingual reasoning performance. Furthermore, a two-step training scheme is introduced to first train to embeded the external capabilities into LLMs and then train the collaborative utilization of the external capabilities and the built-in capabilities in LLMs. Experiments on three multilingual reasoning datasets and a language understanding dataset demonstrate that MindMerger consistently outperforms all baselines, especially in low-resource languages. Without updating the parameters of LLMs, the average accuracy improved by 6.7% and 8.0% across all languages and low-resource languages on the MGSM dataset, respectively.<|reference_end|>
@article{huang2024mindmerger:,
    title={MindMerger: Efficient Boosting LLM Reasoning in non-English Languages},
    author={Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, Fei Yuan},
    journal={arXiv preprint arXiv:2405.17386},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.17386},
    primaryClass={cs.CL cs.AI}
}