Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents
<|reference_start|>Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and Documents: Encoder models trained for the embedding of sentences or short documents have proven useful for tasks such as semantic search and topic modeling. In this paper, we present a version of the SwissBERT encoder model that we specifically fine-tuned for this purpose. SwissBERT contains language adapters for the four national languages of Switzerland -- German, French, Italian, and Romansh -- and has been pre-trained on a large number of news articles in those languages. Using contrastive learning based on a subset of these articles, we trained a fine-tuned version, which we call SentenceSwissBERT. Multilingual experiments on document retrieval and text classification in a Switzerland-specific setting show that SentenceSwissBERT surpasses the accuracy of the original SwissBERT model and of a comparable baseline. The model is openly available for research use.<|reference_end|>
@article{grosjean2024fine-tuning,
    title={Fine-tuning the SwissBERT Encoder Model for Embedding Sentences and
  Documents},
    author={Juri Grosjean and Jannis Vamvas},
    journal={arXiv preprint arXiv:2405.07513},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.07513},
    primaryClass={cs.CL}
}