An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing
<|reference_start|>An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing: We present Expert-Token-Routing, a unified generalist framework that facilitates seamless integration of multiple expert LLMs. Our framework represents expert LLMs as special expert tokens within the vocabulary of a meta LLM. The meta LLM can route to an expert LLM like generating new tokens. Expert-Token-Routing not only supports learning the implicit expertise of expert LLMs from existing instruction dataset but also allows for dynamic extension of new expert LLMs in a plug-and-play manner. It also conceals the detailed collaboration process from the user's perspective, facilitating interaction as though it were a singular LLM. Our framework outperforms various existing multi-LLM collaboration paradigms across benchmarks that incorporate six diverse expert domains, demonstrating effectiveness and robustness in building generalist LLM system via synergizing multiple expert LLMs.<|reference_end|>
@article{chai2024an,
    title={An Expert is Worth One Token: Synergizing Multiple Expert LLMs as
  Generalist via Expert Token Routing},
    author={Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu
  Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, Yang Yang},
    journal={arXiv preprint arXiv:2403.16854},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.16854},
    primaryClass={cs.CL cs.AI}
}