Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study
<|reference_start|>Adaptation of Biomedical and Clinical Pretrained Models to French Long Documents: A Comparative Study: Recently, pretrained language models based on BERT have been introduced for the French biomedical domain. Although these models have achieved state-of-the-art results on biomedical and clinical NLP tasks, they are constrained by a limited input sequence length of 512 tokens, which poses challenges when applied to clinical notes. In this paper, we present a comparative study of three adaptation strategies for long-sequence models, leveraging the Longformer architecture. We conducted evaluations of these models on 16 downstream tasks spanning both biomedical and clinical domains. Our findings reveal that further pre-training an English clinical model with French biomedical texts can outperform both converting a French biomedical BERT to the Longformer architecture and pre-training a French biomedical Longformer from scratch. The results underscore that long-sequence French biomedical models improve performance across most downstream tasks regardless of sequence length, but BERT based models remain the most efficient for named entity recognition tasks.<|reference_end|>
@article{bazoge2024adaptation,
    title={Adaptation of Biomedical and Clinical Pretrained Models to French Long
  Documents: A Comparative Study},
    author={Adrien Bazoge, Emmanuel Morin, Beatrice Daille, Pierre-Antoine
  Gourraud},
    journal={arXiv preprint arXiv:2402.16689},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.16689},
    primaryClass={cs.CL cs.AI}
}