tomaarsen/splade-distilbert-base-uncased-msmarco-mnrl

splade-distilbert-base-uncased trained on MS MARCO triplets

This is a SPLADE Sparse Encoder model finetuned from distilbert/distilbert-base-uncased on the msmarco dataset using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: distilbert/distilbert-base-uncased
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • msmarco
• Language: en
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 256, 'do_lower_case': False}) with MLMTransformer model: DistilBertForMaskedLM 
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("tomaarsen/splade-distilbert-base-uncased-msmarco-mnrl")
# Run inference
queries = [
    "canine ultrasound cost",
]
documents = [
    'VetInfo indicates that any type of canine ultrasound costs anywhere from $300 to $500 depending on what region of the country you live in and whether or not a veterinarian or a technician performs the procedure.',
    '1 Lymphoma, the most common canine cancer, usually requires only chemotherapy and its cost can come up to be around $450 to $500. 2  Osteosarcoma, another type of canine cancer, is usually treated with chemotherapy along with amputation surgery.3  This type of chemotherapy treatment costs approximately $450.nother factor is the type of drugs used in the process. The size of the dog that needs to undergo chemotherapy can also impact the cost. Even a dog very small in size with a single cancerous lesion can cost $200 for chemotherapy, while the same problem on a larger breed could cost more than $1,000 a month.',
    'Plant Life. There are many different plants in the rain forest. Some of the plants include vines, bromeliads, the passion fruit plant and the Victorian water lily. Vines in the rainforest can be as thick as the average human average human body and some can grow to be 3,000 ft long.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[33.7473, 25.6638,  0.2965]])

Evaluation

Metrics

Sparse Information Retrieval

• Datasets: NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with SparseInformationRetrievalEvaluator

Metric	NanoMSMARCO	NanoNFCorpus	NanoNQ	NanoClimateFEVER	NanoDBPedia	NanoFEVER	NanoFiQA2018	NanoHotpotQA	NanoQuoraRetrieval	NanoSCIDOCS	NanoArguAna	NanoSciFact	NanoTouche2020
dot_accuracy@1	0.38	0.38	0.52	0.26	0.76	0.8	0.32	0.9	0.84	0.44	0.12	0.54	0.6531
dot_accuracy@3	0.66	0.54	0.7	0.46	0.88	0.9	0.54	0.96	0.94	0.56	0.44	0.64	0.8367
dot_accuracy@5	0.72	0.58	0.72	0.58	0.88	0.96	0.6	0.96	0.96	0.62	0.54	0.74	0.898
dot_accuracy@10	0.86	0.62	0.78	0.7	0.9	0.98	0.7	0.96	0.96	0.76	0.72	0.78	0.9796
dot_precision@1	0.38	0.38	0.52	0.26	0.76	0.8	0.32	0.9	0.84	0.44	0.12	0.54	0.6531
dot_precision@3	0.22	0.3667	0.2333	0.16	0.6067	0.3067	0.24	0.52	0.36	0.2733	0.1467	0.2333	0.6122
dot_precision@5	0.144	0.328	0.144	0.124	0.584	0.204	0.172	0.324	0.224	0.224	0.108	0.16	0.5796
dot_precision@10	0.086	0.266	0.082	0.094	0.514	0.104	0.106	0.172	0.122	0.162	0.072	0.086	0.5041
dot_recall@1	0.38	0.0218	0.5	0.1317	0.0886	0.7667	0.1621	0.45	0.774	0.0927	0.12	0.495	0.0458
dot_recall@3	0.66	0.0741	0.66	0.229	0.1745	0.8667	0.3618	0.78	0.8853	0.1697	0.44	0.615	0.1279
dot_recall@5	0.72	0.0936	0.68	0.2697	0.2491	0.9333	0.4036	0.81	0.902	0.2297	0.54	0.715	0.1953
dot_recall@10	0.86	0.1197	0.75	0.3687	0.3582	0.9433	0.4941	0.86	0.93	0.3307	0.72	0.76	0.3217
dot_ndcg@10	0.6227	0.3145	0.64	0.2935	0.6321	0.8746	0.3856	0.8341	0.8912	0.3267	0.4019	0.6395	0.5553
dot_mrr@10	0.5469	0.4677	0.6204	0.381	0.8133	0.8663	0.4343	0.9267	0.8917	0.53	0.3024	0.6164	0.7593
dot_map@100	0.5547	0.1298	0.6045	0.2226	0.4903	0.8476	0.321	0.783	0.874	0.2468	0.3153	0.6051	0.4021
query_active_dims	23.62	18.78	26.94	52.7	22.98	41.62	22.68	43.14	21.78	40.18	142.1	54.24	20.5306
query_sparsity_ratio	0.9992	0.9994	0.9991	0.9983	0.9992	0.9986	0.9993	0.9986	0.9993	0.9987	0.9953	0.9982	0.9993
corpus_active_dims	86.9286	164.7944	109.7591	152.8175	103.2382	154.9832	89.7392	108.2432	24.8411	145.2197	133.5271	172.6595	106.1772
corpus_sparsity_ratio	0.9972	0.9946	0.9964	0.995	0.9966	0.9949	0.9971	0.9965	0.9992	0.9952	0.9956	0.9943	0.9965

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "msmarco",
          "nfcorpus",
          "nq"
      ]
  }
  

Metric	Value
dot_accuracy@1	0.4267
dot_accuracy@3	0.6333
dot_accuracy@5	0.6733
dot_accuracy@10	0.7533
dot_precision@1	0.4267
dot_precision@3	0.2733
dot_precision@5	0.2053
dot_precision@10	0.1447
dot_recall@1	0.3006
dot_recall@3	0.4647
dot_recall@5	0.4979
dot_recall@10	0.5766
dot_ndcg@10	0.5258
dot_mrr@10	0.545
dot_map@100	0.4297
query_active_dims	23.1133
query_sparsity_ratio	0.9992
corpus_active_dims	113.3954
corpus_sparsity_ratio	0.9963

Sparse Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with SparseNanoBEIREvaluator with these parameters:

  {
      "dataset_names": [
          "climatefever",
          "dbpedia",
          "fever",
          "fiqa2018",
          "hotpotqa",
          "msmarco",
          "nfcorpus",
          "nq",
          "quoraretrieval",
          "scidocs",
          "arguana",
          "scifact",
          "touche2020"
      ]
  }
  

Metric	Value
dot_accuracy@1	0.5318
dot_accuracy@3	0.6967
dot_accuracy@5	0.7506
dot_accuracy@10	0.823
dot_precision@1	0.5318
dot_precision@3	0.3291
dot_precision@5	0.2554
dot_precision@10	0.1823
dot_recall@1	0.3099
dot_recall@3	0.4649
dot_recall@5	0.5186
dot_recall@10	0.6013
dot_ndcg@10	0.5702
dot_mrr@10	0.6274
dot_map@100	0.4921
query_active_dims	40.8998
query_sparsity_ratio	0.9987
corpus_active_dims	112.7496
corpus_sparsity_ratio	0.9963

Training Details

Training Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 90,000 training samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative
  type	string	string	string
  details	min: 4 tokens mean: 9.1 tokens max: 30 tokens	min: 24 tokens mean: 79.91 tokens max: 218 tokens	min: 22 tokens mean: 77.09 tokens max: 256 tokens

• Samples:

  query	positive	negative
  when do manga complete editions	Volumes 9 and 10 the Sailor Moon Manga Complete Editions are now out. Last week, March 25th, volumes 9 and 10 of the Sailor Moon Manga Complete Editions were released in Japan. Volume 9 features Endymion and Serenity on the cover while volume 10 features all 10 Sailor Guardians.	Destiny: The Taken King will be released in standard download, collector’s edition download, and both “Collector’s” and “Legendary” game disc editions on September 15, 2015.
  the define of homograph	LINK / CITE ADD TO WORD LIST. noun. The definition of a homograph is a word that is spelled like another word or other words, but has a different meaning and sometimes sounds different. An example of a homograph is evening, which is the time of day after the sun has set or making something level or flat.	As verbs the difference between describe and define. is that describe is to represent in words while define is to determine. As a noun define is. (computing
  what is a cv in resume writing	Curriculum Vitae (CV) is Latin for “course of life.” In contrast, resume is French for “summary.” Both CVs & Resumes: 1 Are tailored for the specific job/company you are applying to. 2 Should represent you as the best qualified candidate. 3 Are used to get you an interview. Do not usually include personal interests.	Resume Samples » Resume Objective » Legal Resume Objective » Legal Assistant Resume Objective. Job Description: Legal assistant is responsible to manage and handle various activities of legal department. Preparing legal documents such as contracts, wills and appeals.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "lambda_corpus": 0.001,
      "lambda_query": 5e-05
  }
  

Evaluation Dataset

msmarco

• Dataset: msmarco at 9e329ed
• Size: 10,000 evaluation samples
• Columns: query, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	query	positive	negative
  type	string	string	string
  details	min: 5 tokens mean: 9.24 tokens max: 53 tokens	min: 13 tokens mean: 80.9 tokens max: 204 tokens	min: 17 tokens mean: 78.24 tokens max: 234 tokens

• Samples:

  query	positive	negative
  the largest vietnamese population in the united states is in	The largest number of Vietnamese outside Vietnam is in Orange County, California (184,153, or 6.1 percent of the county's population), followed by Los Angeles and Santa Clara counties; the three counties accounted for 26 percent of the Vietnamese immigrant population in the United States.	Population by Place in the United States There are 29,257 places in the United States. This section compares Hibbing to the 50 most populous places in the United States. The least populous of the compared places has a population of 371,267.
  how many calories in a tablespoon of flaxseed	Calorie Content. A 2-tablespoon serving of ground flaxseed has about 75 calories, according to the U.S. Department of Agriculture. These calories consist of 2.6 grams of protein, 4 grams of carbohydrates -- almost all of which is fiber -- and 6 grams of fat.	You can also use flaxseed meal to replace an egg. Use 1 tablespoon flaxseed meal and 3 tablespoons water. You can replace up to two eggs in a recipe in this manner, but do not use flaxseed meal as an egg replacement if you are already using it as an oil replacement. Flaxseed has many health benefits.
  who wrote the house of seven gables	The author of The House Of Seven Gables is Nathaniel Hawthorne.	Abigail Adams Wrote To John In 1776: Remember The Ladies Or We'll Rebel Adams wrote a feminist letter to her husband just before U.S. independence.

• Loss: SpladeLoss with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score')",
      "lambda_corpus": 0.001,
      "lambda_query": 5e-05
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss	Validation Loss	NanoMSMARCO_dot_ndcg@10	NanoNFCorpus_dot_ndcg@10	NanoNQ_dot_ndcg@10	NanoBEIR_mean_dot_ndcg@10	NanoClimateFEVER_dot_ndcg@10	NanoDBPedia_dot_ndcg@10	NanoFEVER_dot_ndcg@10	NanoFiQA2018_dot_ndcg@10	NanoHotpotQA_dot_ndcg@10	NanoQuoraRetrieval_dot_ndcg@10	NanoSCIDOCS_dot_ndcg@10	NanoArguAna_dot_ndcg@10	NanoSciFact_dot_ndcg@10	NanoTouche2020_dot_ndcg@10
0.0178	100	173.8874	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0356	200	11.8803	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0533	300	1.0264	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0711	400	0.3923	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.0889	500	0.32	0.2369	0.4932	0.3001	0.5819	0.4584	-	-	-	-	-	-	-	-	-	-
0.1067	600	0.2483	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1244	700	0.28	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1422	800	0.2095	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.16	900	0.2093	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.1778	1000	0.1636	0.1898	0.6051	0.2845	0.6124	0.5006	-	-	-	-	-	-	-	-	-	-
0.1956	1100	0.1661	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2133	1200	0.1964	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2311	1300	0.1937	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2489	1400	0.1771	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.2667	1500	0.1643	0.1549	0.5868	0.3176	0.5769	0.4938	-	-	-	-	-	-	-	-	-	-
0.2844	1600	0.1987	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3022	1700	0.178	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.32	1800	0.1227	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3378	1900	0.1478	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3556	2000	0.1502	0.1563	0.6249	0.3309	0.6088	0.5215	-	-	-	-	-	-	-	-	-	-
0.3733	2100	0.1623	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.3911	2200	0.1703	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4089	2300	0.1804	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4267	2400	0.121	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4444	2500	0.1451	0.1325	0.5620	0.3233	0.6197	0.5017	-	-	-	-	-	-	-	-	-	-
0.4622	2600	0.1609	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.48	2700	0.1415	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.4978	2800	0.1555	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5156	2900	0.1581	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5333	3000	0.1351	0.1546	0.5901	0.3187	0.6299	0.5129	-	-	-	-	-	-	-	-	-	-
0.5511	3100	0.1308	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5689	3200	0.1313	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.5867	3300	0.1248	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6044	3400	0.1295	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6222	3500	0.1398	0.1449	0.6096	0.3285	0.5975	0.5119	-	-	-	-	-	-	-	-	-	-
0.64	3600	0.1105	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6578	3700	0.0911	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6756	3800	0.1683	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.6933	3900	0.1202	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7111	4000	0.135	0.1592	0.5989	0.3109	0.6460	0.5186	-	-	-	-	-	-	-	-	-	-
0.7289	4100	0.1205	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7467	4200	0.1432	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7644	4300	0.105	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.7822	4400	0.1028	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8	4500	0.1386	0.1383	0.5859	0.3084	0.6276	0.5073	-	-	-	-	-	-	-	-	-	-
0.8178	4600	0.1068	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8356	4700	0.1262	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8533	4800	0.1182	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8711	4900	0.1331	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.8889	5000	0.1436	0.1279	0.6261	0.3136	0.6314	0.5237	-	-	-	-	-	-	-	-	-	-
0.9067	5100	0.1182	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9244	5200	0.1379	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9422	5300	0.1343	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.96	5400	0.1475	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
0.9778	5500	0.0988	0.1311	0.6227	0.3145	0.64	0.5258	-	-	-	-	-	-	-	-	-	-
0.9956	5600	0.1072	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
-1	-1	-	-	0.6227	0.3145	0.6400	0.5702	0.2935	0.6321	0.8746	0.3856	0.8341	0.8912	0.3267	0.4019	0.6395	0.5553

• The bold row denotes the saved checkpoint.

Environmental Impact

Carbon emissions were measured using CodeCarbon.

• Energy Consumed: 0.138 kWh
• Carbon Emitted: 0.054 kg of CO2
• Hours Used: 0.458 hours

Training Hardware

• On Cloud: No
• GPU Model: 1 x NVIDIA GeForce RTX 3090
• CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM Size: 31.78 GB

Framework Versions

• Python: 3.11.6
• Sentence Transformers: 4.2.0.dev0
• Transformers: 4.52.4
• PyTorch: 2.7.1+cu126
• Accelerate: 1.5.1
• Datasets: 2.21.0
• Tokenizers: 0.21.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
    }