Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise
<|reference_start|>Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise: Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities. However, we observed that despite improvements in specific domain knowledge, the performance of medical LLM in long-context understanding has significantly declined, especially compared to general language models with similar parameters. The purpose of this study is to investigate the phenomenon of reduced performance in understanding long-context in medical LLM. We designed a series of experiments to conduct open-book professional knowledge exams on all models to evaluate their ability to read long-context. By adjusting the proportion and quantity of general data and medical data in the process of fine-tuning, we can determine the best data composition to optimize the professional model and achieve a balance between long-context performance and specific domain knowledge.<|reference_end|>
@article{yang2024fine-tuning,
    title={Fine-Tuning Medical Language Models for Enhanced Long-Contextual
  Understanding and Domain Expertise},
    author={Qimin Yang, Rongsheng Wang, Jiexin Chen, Runqi Su, Tao Tan},
    journal={arXiv preprint arXiv:2407.11536},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.11536},
    primaryClass={cs.CL cs.AI}
}