QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
<|reference_start|>QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling: Large Language models (LLMs) have brought about substantial advancements in the field of Question Answering (QA) systems. These models do remarkably well in addressing intricate inquiries in a variety of disciplines. However, because of domain-specific vocabulary, complex technological concepts, and the requirement for exact responses applying LLMs to specialized sectors like telecommunications presents additional obstacles. GPT-3.5 has been used in recent work, to obtain noteworthy accuracy for telecom-related questions in a Retrieval Augmented Generation (RAG) framework. Notwithstanding these developments, the practical use of models such as GPT-3.5 is restricted by their proprietary nature and high computing demands. This paper introduces QMOS, an innovative approach which uses a Question-Masked loss and Option Shuffling trick to enhance the performance of LLMs in answering Multiple-Choice Questions in the telecommunications domain. Our focus was on using opensource, smaller language models (Phi-2 and Falcon-7B) within an enhanced RAG framework. Our multi-faceted approach involves several enhancements to the whole LLM-RAG pipeline of finetuning, retrieval, prompt engineering and inference. Our approaches significantly outperform existing results, achieving accuracy improvements from baselines of 24.70% to 49.30% with Falcon-7B and from 42.07% to 84.65% with Phi-2.<|reference_end|>
@article{guda2024qmos:,
    title={QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and
  Option Shuffling},
    author={Blessed Guda, Gabrial Zencha A., Lawrence Francis and Carlee Joe-Wong},
    journal={arXiv preprint arXiv:2409.14175},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14175},
    primaryClass={cs.CL cs.AI cs.LG}
}