title: LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits
    Siamese-BLOOM
summary: Text embeddings are useful features for several NLP applications, such as
  sentence similarity, text clustering, and semantic search. In this paper, we
  present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
  Siamese-BLOOM, a multilingual large language model optimized to produce
  semantically meaningful word embeddings. The innovation is threefold. First, we
  cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
  adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
  Third, we apply a Siamese architecture on BLOOM model with a contrastive
  objective to ease the multi-lingual labeled data scarcity. The experiment
  results show the quality of learned embeddings from LACoS-BLOOM is proportional
  to the number of model parameters and the amount of unlabeled training data.
  With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
  billion parameters end-to-end on a single GPU machine with 32GB memory.
  Compared to previous solution Sentence-BERT, we achieve significant improvement
  on both English and multi-lingual STS tasks.
paper_authors._id: 645da48bf5760d1530d6e297
paper_authors.name: Wen-Yu Hua
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-05-12T02:29:31.587Z
paper_authors._id: 64091cb4bf3e9a4bb2e30b75
paper_authors.avatarUrl: /avatars/96bb4a5abc26684d62f3d16339774c28.svg
paper_authors.fullname: Wendy Kung
paper_authors.type: user
paper_authors.user: hhuietkung
paper_authors._id: 645da48bf5760d1530d6e298
paper_authors.name: Brian Williams
paper_authors._id: 645da48bf5760d1530d6e299
paper_authors.name: Davood Shamsi
paper_title: LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits
    Siamese-BLOOM
paper_summary: Text embeddings are useful features for several NLP applications, such as
  sentence similarity, text clustering, and semantic search. In this paper, we
  present a Low-rank Adaptation with a Contrastive objective on top of 8-bit
  Siamese-BLOOM, a multilingual large language model optimized to produce
  semantically meaningful word embeddings. The innovation is threefold. First, we
  cast BLOOM weights to 8-bit values. Second, we fine-tune BLOOM with a scalable
  adapter (LoRA) and 8-bit Adam optimizer for sentence similarity classification.
  Third, we apply a Siamese architecture on BLOOM model with a contrastive
  objective to ease the multi-lingual labeled data scarcity. The experiment
  results show the quality of learned embeddings from LACoS-BLOOM is proportional
  to the number of model parameters and the amount of unlabeled training data.
  With the parameter efficient fine-tuning design, we are able to run BLOOM 7.1
  billion parameters end-to-end on a single GPU machine with 32GB memory.
  Compared to previous solution Sentence-BERT, we achieve significant improvement
  on both English and multi-lingual STS tasks.
paper_ai_summary: LACoS-BLOOM, an 8-bit Low-rank Adaptation with Contrastive objective of Siamese-BLOOM, improves text embeddings quality and efficiency in NLP tasks.
paper_ai_keywords: Low-rank Adaptation Contrastive objective 8-bit Siamese-BLOOM multilingual large language model semantic word embeddings sentence similarity classification scalable adapter LoRA 8-bit Adam optimizer Siamese architecture multi-lingual labeled data scarcity parameter efficient fine-tuning LACoS-BLOOM Sentence-BERT English and multi-lingual STS tasks