iampanda/zpoint_large_embedding_zh

ZPoint Large Embedding for Chinese

• [2024-06-04] Release zpoint_large_embedding_zh, and upload model weight to huggingface
• [2024-06-05] Add training details

Training Details

Base Model

1. We chose Stella as our base model.

Training Data

1. Hard negative samping

• For retrieval task, We sampled 10 hard negative passages/answers from top50-top200 related passages/answers for each query.
• For classification/clustering tasks, we sampled 5 hard negative samples from other classes/cluster for each sample.
• For classification/clustering tasks, we also used the category names of each class and cluster as positive and negative samples.

2. Data synthesis by LLM (ZPoint-72B)

• For retrieval tasks, we used LLM to rewrite each query, generating five different rewritten results.
• For retrieval tasks, we also generated five new queries for some documents by LLM.
• For non-retrieval tasks, we used LLM to rewrite the queries, generating five rewritten results for each query.
• Finally, total amount of synthesized data is about 30 million.

3. Collect more data for retrieval-type tasks

• miracl/miracl
• FreedomIntelligence/Huatuo26M-Lite
• PaddlePaddle/dureader_robust C-MTEB test filtered
• THUIR/T2Ranking C-MTEB test filtered
• Shitao/bge-reranker-data
• Shitao/MLDR
• ...

We constructed a dataset of approximately 100 million training samples through collection, machine translation, and LLM synthesis. This dataset includes data from various fields such as healthcare, law, electricity, automotive, and 3C (Consumer Electronics).

Training loss

1. Multi-Task loss like Piccolo
2. Matryoshka Representation Learning

Example

from sentence_transformers import SentenceTransformer
sentences1 = ["这个产品真垃圾"]
sentences2 = ["我太喜欢这个产品了"]
model = SentenceTransformer('iampanda/zpoint_large_embedding_zh')
embeddings_1 = model.encode(sentences1, normalize_embeddings=True)
embeddings_2 = model.encode(sentences2, normalize_embeddings=True)
similarity = embeddings_1 @ embeddings_2.T
print(similarity)