Efficiently Distilling LLMs for Edge Applications
<|reference_start|>Efficiently Distilling LLMs for Edge Applications: Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.<|reference_end|>
@article{kundu2024efficiently,
    title={Efficiently Distilling LLMs for Edge Applications},
    author={Achintya Kundu, Fabian Lim, Aaron Chew, Laura Wynter, Penny Chong,
  Rhui Dih Lee},
    journal={arXiv preprint arXiv:2404.01353},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.01353},
    primaryClass={cs.LG cs.AI cs.CL}
}