Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
<|reference_start|>Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts: We present a toolkit for creating low-cost Mixture-of-Domain-Experts (MOE) from trained models. The toolkit can be used for creating a mixture from models or from adapters. We perform extensive tests and offer guidance on defining the architecture of the resulting MOE using the toolkit. A public repository is available.<|reference_end|>
@article{lee2024flexible,
    title={Flexible and Effective Mixing of Large Language Models into a Mixture of
  Domain Experts},
    author={Rhui Dih Lee and Laura Wynter and Raghu Kiran Ganti},
    journal={arXiv preprint arXiv:2408.17280},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.17280},
    primaryClass={cs.AI cs.CL}
}