Extracting Sentence Embeddings from Pretrained Transformer Models
<|reference_start|>Extracting Sentence Embeddings from Pretrained Transformer Models: Background/introduction: Pre-trained transformer models shine in many natural language processing tasks and therefore are expected to bear the representation of the input sentence or text meaning. These sentence-level embeddings are also important in retrieval-augmented generation. But do commonly used plain averaging or prompt templates surface it enough?   Methods: Given 110M parameters BERT's hidden representations from multiple layers and multiple tokens we tried various ways to extract optimal sentence representations. We tested various token aggregation and representation post-processing techniques. We also tested multiple ways of using a general Wikitext dataset to complement BERTs sentence representations. All methods were tested on 8 Semantic Textual Similarity (STS), 6 short text clustering, and 12 classification tasks. We also evaluated our representation-shaping techniques on other static models, including random token representations.   Results: Proposed representation extraction methods improved the performance on STS and clustering tasks for all models considered. Very high improvements for static token-based models, especially random embeddings for STS tasks almost reach the performance of BERT-derived representations.   Conclusions: Our work shows that for multiple tasks simple baselines with representation shaping techniques reach or even outperform more complex BERT-based models or are able to contribute to their performance.<|reference_end|>
@article{stankevičius2024extracting,
    title={Extracting Sentence Embeddings from Pretrained Transformer Models},
    author={Lukas Stankeviv{c}ius and Mantas Lukov{s}eviv{c}ius},
    journal={arXiv preprint arXiv:2408.08073},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.08073},
    primaryClass={cs.CL cs.IR cs.LG stat.ML}
}