NeuML/pubmedbert-base-splade

PubMedBERT SPLADE

This is a SPLADE Sparse Encoder model finetuned from PubMedBERT-base using sentence-transformers. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

The training dataset was generated using a random sample of PubMed title-abstract pairs along with similar title pairs.

PubMedBERT SPLADE produces higher quality sparse embeddings than generalized models for medical literature. Further fine-tuning for a medical subdomain will result in even better performance.

Usage (txtai)

This model can be used to build embeddings databases with txtai for semantic search and/or as a knowledge source for retrieval augmented generation (RAG).

Note: txtai 8.7.0+ is required for sparse vector scoring support

import txtai

embeddings = txtai.Embeddings(
  sparse="neuml/pubmedbert-base-splade",
  content=True
)
embeddings.index(documents())

# Run a query
embeddings.search("query to run")

Usage (Sentence-Transformers)

Alternatively, the model can be loaded with sentence-transformers.

from sentence_transformers import SpladeEncoder
sentences = ["This is an example sentence", "Each sentence is converted"]

model = SpladeEncoder("neuml/pubmedbert-base-splade")
embeddings = model.encode(sentences)
print(embeddings)

Evaluation Results

Performance of this model compared to the top base models on the MTEB leaderboard is shown below. A popular smaller model was also evaluated along with the most downloaded PubMed similarity model on the Hugging Face Hub.

The following datasets were used to evaluate model performance.

• PubMed QA
  • Subset: pqa_labeled, Split: train, Pair: (question, long_answer)
• PubMed Subset
  • Split: test, Pair: (title, text)
• PubMed Summary
  • Subset: pubmed, Split: validation, Pair: (article, abstract)

Evaluation results are shown below. The Pearson correlation coefficient is used as the evaluation metric.

Model	PubMed QA	PubMed Subset	PubMed Summary	Average
all-MiniLM-L6-v2	90.40	95.92	94.07	93.46
bge-base-en-v1.5	91.02	95.82	94.49	93.78
gte-base	92.97	96.90	96.24	95.37
pubmedbert-base-embeddings	93.27	97.00	96.58	95.62
pubmedbert-base-splade	90.76	96.20	95.87	94.28
S-PubMedBert-MS-MARCO	90.86	93.68	93.54	92.69

While this model was't the highest scoring model using the Pearson metric, it does well when measured by Spearman rank correlation coefficient.

Model	PubMed QA	PubMed Subset	PubMed Summary	Average
all-MiniLM-L6-v2	85.77	86.52	86.32	86.20
bge-base-en-v1.5	85.71	86.58	86.35	86.21
gte-base	86.44	86.60	86.55	86.53
pubmedbert-base-embeddings	86.29	86.57	86.47	86.44
pubmedbert-base-splade	86.80	89.12	88.60	88.17
S-PubMedBert-MS-MARCO	85.71	86.37	86.13	86.07

This indicates that the SPLADE model may do a better job of calculating scores/rankings in the correct direction.

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

More Information

The training data for this model is the same as described in this article. See this article for more on the training scripts.