Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples
<|reference_start|>Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples: Decoder-based large language models (LLMs) have shown high performance on many tasks in natural language processing. This is also true for sentence embedding learning, where a decoder-based model, PromptEOL, has achieved the best performance on semantic textual similarity (STS) tasks. However, PromptEOL requires a manually annotated natural language inference (NLI) dataset for fine-tuning. We aim to improve sentence embeddings without using large manually annotated datasets by automatically generating an NLI dataset with an LLM and using it for fine-tuning of PromptEOL. To achieve this, we explore methods of data generation suitable for sentence embedding learning in this study. Specifically, we will focus on automatic dataset generation through few-shot learning and explore the appropriate methods to leverage few-shot examples. Experimental results on the STS tasks demonstrate that our approach outperforms existing models in settings without large manually annotated datasets.<|reference_end|>
@article{sato2024improving,
    title={Improving Sentence Embeddings with Automatic Generation of Training Data
  Using Few-shot Examples},
    author={Soma Sato, Hayato Tsukagoshi, Ryohei Sasano, Koichi Takeda},
    journal={arXiv preprint arXiv:2402.15132},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.15132},
    primaryClass={cs.CL cs.LG}
}