Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field
<|reference_start|>Domain-Specific Pretraining of Language Models: A Comparative Study in the Medical Field: There are many cases where LLMs are used for specific tasks in a single domain. These usually require less general, but more domain-specific knowledge. Highly capable, general-purpose state-of-the-art language models like GPT-4 or Claude-3-opus can often be used for such tasks, but they are very large and cannot be run locally, even if they were not proprietary. This can be a problem when working with sensitive data. This paper focuses on domain-specific and mixed-domain pretraining as potentially more efficient methods than general pretraining for specialized language models. We will take a look at work related to domain-specific pretraining, specifically in the medical area, and compare benchmark results of specialized language models to general-purpose language models.<|reference_end|>
@article{kerner2024domain-specific,
    title={Domain-Specific Pretraining of Language Models: A Comparative Study in
  the Medical Field},
    author={Tobias Kerner},
    journal={arXiv preprint arXiv:2407.14076},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.14076},
    primaryClass={cs.LG cs.AI cs.CL}
}