title: Improving Text Embeddings for Smaller Language Models Using Contrastive
    Fine-tuning
summary: While Large Language Models show remarkable performance in natural language
  understanding, their resource-intensive nature makes them less accessible. In
  contrast, smaller language models such as MiniCPM offer more sustainable
  scalability, but often underperform without specialized optimization. In this
  paper, we explore the enhancement of smaller language models through the
  improvement of their text embeddings. We select three language models, MiniCPM,
  Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our
  results demonstrate that this fine-tuning method enhances the quality of text
  embeddings for all three models across various benchmarks, with MiniCPM showing
  the most significant improvements of an average 56.33\% performance gain. The
  contrastive fine-tuning code is publicly available at
  https://github.com/trapoom555/Language-Model-STS-CFT.
paper_authors._id: 66ac43e305560e24ec2701e0
paper_authors.name: Trapoom Ukarapol
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:47:33.876Z
paper_authors._id: 661cae1bd7c07238c295352b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/661cae1bd7c07238c295352b/_ZYxfOMyrUlkxSjx0WBwG.jpeg
paper_authors.fullname: Trapoom Ukarapol
paper_authors.type: user
paper_authors.user: trapoom555
paper_authors._id: 66ac43e305560e24ec2701e1
paper_authors.name: Zhicheng Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:47:44.296Z
paper_authors._id: 652542861e9db26e407aa1fc
paper_authors.avatarUrl: /avatars/4c47ef4564f498a7f34b4a17a1e209a8.svg
paper_authors.fullname: Lee Zhicheng
paper_authors.type: user
paper_authors.user: ZhiCheng0326
paper_authors._id: 66ac43e305560e24ec2701e2
paper_authors.name: Amy Xin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-02T08:47:50.305Z
paper_authors._id: 660bf98c3336a7e128a0e918
paper_authors.avatarUrl: /avatars/3e3f2886bd4a730ec19b13aecc99279f.svg
paper_authors.fullname: Amy Xin
paper_authors.type: user
paper_authors.user: amyxx2001
paper_title: Improving Text Embeddings for Smaller Language Models Using Contrastive
    Fine-tuning
paper_summary: While Large Language Models show remarkable performance in natural language
  understanding, their resource-intensive nature makes them less accessible. In
  contrast, smaller language models such as MiniCPM offer more sustainable
  scalability, but often underperform without specialized optimization. In this
  paper, we explore the enhancement of smaller language models through the
  improvement of their text embeddings. We select three language models, MiniCPM,
  Phi-2, and Gemma, to conduct contrastive fine-tuning on the NLI dataset. Our
  results demonstrate that this fine-tuning method enhances the quality of text
  embeddings for all three models across various benchmarks, with MiniCPM showing
  the most significant improvements of an average 56.33\% performance gain. The
  contrastive fine-tuning code is publicly available at
  https://github.com/trapoom555/Language-Model-STS-CFT.
paper_ai_keywords: Large Language Models natural language understanding resource-intensive MiniCPM sustainable scalability specialized optimization text embeddings contrastive fine-tuning NLI dataset benchmarks performance gain