spl4shedEdu/mpnet_ESB

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'moor brown gb 2014 audi a5s5 cabriolet south africa market body middle armrest front pr6e3gb if period 0911 gb automotive',
    'moor brown gb 2013 audi a4 allroad quattro canada market body middle armrest front pr6e3gb if period 1111 gb automotive',
    'smart racing products smartstrings 4wheel alignment set 1996 bmw 318i base convertible suspension tools page 1 automotive',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 269,761 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 15 tokens mean: 69.38 tokens max: 384 tokens	min: 11 tokens mean: 70.41 tokens max: 384 tokens

• Samples:

  anchor	positive
  2008 dodge ram 2500 laramie used white for sale in euless tx dodge automotive mileage 17383 exterior color arctic white transmission 7speed automatic wod engine intercooled turbo diesel i4 21 l131 stock 4gp307369 wdzpe7dd4gp307369 mercedesbenz used sprinter passenger vans mileage 148438 exterior color white transmission 4speed hd automatic wod 4l65e engine gas v8 60l364 stock x4h100607 5grgn23ux4h100607 hummer used h2 mileage 34383 exterior color white satin pearl transmission 8speed automatic wmanual shift engine gas v8 50l307 stock 4du072090 kmhgh4jh4du072090 hyundai used equus mileage 107520 exterior color bright white clearcoat transmission 6speed automatic engine gas i4 24l144 stock xcn207909 1c3ccbbbxcn207909 chrysler used 200 mileage 59582 exterior color bright white clearcoat transmission 6speed automatic engine gas i4 24l144 stock 3dn554452 1c3ccbbb3dn554452 chrysler used 200 mileage 76681 exterior color taffeta white transmission 5speed automatic engine gas i4 24l144 stock 5ag705280 3czre3h35ag705280 honda used crv mileage 99896 exterior color premium white pearl transmission 5speed automatic wod engine gas i4 24l144 stock 6xac03652 jh4cu2f6xac036527 acura used tsx mileage 125522 exterior color crystal white transmission 1speed ecvt engine gaselectric v6 33l202 stock x82851820 jtjgw31ux82851820 lexus used rx 400h mileage 36797 exterior color clear white transmission 6speed automatic wod engine regular unleaded i4 24 l144 stock 0h7070128 kndpm3ac0h7070128 kia used sportage mileage 77058 exterior color bright white clearcoat transmission 5speed automatic engine gas v6 36l220 stock 0dh649637 2c3cdyag0dh649637 dodge used challenger mileage 39319 exterior color arctic white transmission 7speed automatic engine gas v6 30l183 stock 8br178298 wddgf8bb8br178298 mercedesbenz used cclass mileage 64038 exterior color white gold clearcoat transmission 5speed automatic wod engine gas v8 57l345 stock 7bs615073 1d7rb1ct7bs615073 ram used 1500 mileage 69807 exterior color bright white clearcoat transmission 4speed automatic engine gas v8 57l345 stock 9cs228465 1c6rd6ft9cs228465 ram used 1500 mileage 14918 exterior color pearl white transmission 1speed cvt wod engine regular unleaded i4 25 l152 stock xgp686433 knmat2mtxgp686433 nissan used rogue mileage 39730 exterior color fuji white transmission 6speed automatic engine gas i6 32l195 stock xch305122 salfr2bnxch305122 land rover used lr2 mileage 34618 exterior color snow white pearl transmission 6speed automatic wod engine regular unleaded v6 33 l204 stock 6fg577726 5xykw4a76fg577726 kia used sorento mileage 169861 exterior color oxford white transmission 6speed automatic engine turbocharged diesel v8 67l406 stock 1bea21901 1ft7w2bt1bea21901 ford used super duty f250 srw mileage 21309 exterior color pure red transmission 8speed cvt wod engine intercooled turbo premium unleaded h4 20 l122 stock 3g8807653 jf1va1e63g8807653 subaru used wrx mileage 61401 exterior color white gold clearcoat transmission automatic engine gas v8 57l345 stock 5bg609448 3d7tt2ct5bg609448 ram used 2500 mileage 28253 exterior color bright white clearcoat transmission 8speed automatic wod engine regular unleaded v6 36 l220 stock 2ec478489 1c4rjfbg2ec478489 jeep used grand cherokee mileage 33929 exterior color bright white clearcoat transmission 6speed automatic engine gas v8 57l345 stock xdc653013 1c4sdjctxdc653013 dodge used durango mileage 178293 exterior color bright white clearcoat transmission automatic engine diesel i6 67l409 stock 3d3ks29a dodge used ram 2500 mileage 107825 exterior color diamond white metallic transmission 7speed automatic wmanual shift engine turbocharged gas v8 46l285 stock 9da114638 4jgdf7ce9da114638 mercedesbenz used glclass mileage 17831 exterior color cirrus white transmission 7speed autoshift	2008 dodge ram 2500 laramie used white for sale in euless tx dodge automotive mileage 17383 exterior color arctic white transmission 7speed automatic wod engine intercooled turbo diesel i4 21 l131 stock 4gp307369 wdzpe7dd4gp307369 mercedesbenz used sprinter passenger vans mileage 148438 exterior color white transmission 4speed hd automatic wod 4l65e engine gas v8 60l364 stock x4h100607 5grgn23ux4h100607 hummer used h2 mileage 34383 exterior color white satin pearl transmission 8speed automatic wmanual shift engine gas v8 50l307 stock 4du072090 kmhgh4jh4du072090 hyundai used equus mileage 107520 exterior color bright white clearcoat transmission 6speed automatic engine gas i4 24l144 stock xcn207909 1c3ccbbbxcn207909 chrysler used 200 mileage 59582 exterior color bright white clearcoat transmission 6speed automatic engine gas i4 24l144 stock 3dn554452 1c3ccbbb3dn554452 chrysler used 200 mileage 76681 exterior color taffeta white transmission 5speed automatic engine gas i4 24l144 stock 5ag705280 3czre3h35ag705280 honda used crv mileage 99896 exterior color premium white pearl transmission 5speed automatic wod engine gas i4 24l144 stock 6xac03652 jh4cu2f6xac036527 acura used tsx mileage 125522 exterior color crystal white transmission 1speed ecvt engine gaselectric v6 33l202 stock x82851820 jtjgw31ux82851820 lexus used rx 400h mileage 36797 exterior color clear white transmission 6speed automatic wod engine regular unleaded i4 24 l144 stock 0h7070128 kndpm3ac0h7070128 kia used sportage mileage 77058 exterior color bright white clearcoat transmission 5speed automatic engine gas v6 36l220 stock 0dh649637 2c3cdyag0dh649637 dodge used challenger mileage 39319 exterior color arctic white transmission 7speed automatic engine gas v6 30l183 stock 8br178298 wddgf8bb8br178298 mercedesbenz used cclass mileage 64038 exterior color white gold clearcoat transmission 5speed automatic wod engine gas v8 57l345 stock 7bs615073 1d7rb1ct7bs615073 ram used 1500 mileage 69807 exterior color bright white clearcoat transmission 4speed automatic engine gas v8 57l345 stock 9cs228465 1c6rd6ft9cs228465 ram used 1500 mileage 14918 exterior color pearl white transmission 1speed cvt wod engine regular unleaded i4 25 l152 stock xgp686433 knmat2mtxgp686433 nissan used rogue mileage 39730 exterior color fuji white transmission 6speed automatic engine gas i6 32l195 stock xch305122 salfr2bnxch305122 land rover used lr2 mileage 34618 exterior color snow white pearl transmission 6speed automatic wod engine regular unleaded v6 33 l204 stock 6fg577726 5xykw4a76fg577726 kia used sorento mileage 169861 exterior color oxford white transmission 6speed automatic engine turbocharged diesel v8 67l406 stock 1bea21901 1ft7w2bt1bea21901 ford used super duty f250 srw mileage 21309 exterior color pure red transmission 8speed cvt wod engine intercooled turbo premium unleaded h4 20 l122 stock 3g8807653 jf1va1e63g8807653 subaru used wrx mileage 61401 exterior color white gold clearcoat transmission automatic engine gas v8 57l345 stock 5bg609448 3d7tt2ct5bg609448 ram used 2500 mileage 28253 exterior color bright white clearcoat transmission 8speed automatic wod engine regular unleaded v6 36 l220 stock 2ec478489 1c4rjfbg2ec478489 jeep used grand cherokee mileage 33929 exterior color bright white clearcoat transmission 6speed automatic engine gas v8 57l345 stock xdc653013 1c4sdjctxdc653013 dodge used durango mileage 178293 exterior color bright white clearcoat transmission automatic engine diesel i6 67l409 stock 3d3ks29a dodge used ram 2500 mileage 107825 exterior color diamond white metallic transmission 7speed automatic wmanual shift engine turbocharged gas v8 46l285 stock 9da114638 4jgdf7ce9da114638 mercedesbenz used glclass mileage 17831 exterior color cirrus white transmission 7speed autoshift
  pacon fade resistant neon poster board us pacon officesupplycomus officeproducts	pacon fade resistant neon poster board us pacon officesupplycomus officeproducts
  2015 bmw 535i xdrive sedan texas 76132 2015 535 lease special promotion on a for 87681 per month fort worth automotive	2015 bmw 535i xdrive sedan florida 34787 2015 535 xdriv lease special promotion on sedan a for 76289 per month winter garden automotive listing number car bmw 535i xdrive sedan condition outstanding transmission automatic vin wba5b3c59fd540941 down payment 000 down engine 6 cylinder fuel type gasoline doors four door drivetrain four wheel drive rear brake width 09 emergency braking preparation emergency braking preparation brake drying brake drying front brake width 14 rear brake type ventilated disc electronic brakeforce distribution electronic brakeforce distribution front brake diameter 137 front brake type ventilated disc antilock braking system 4wheel abs braking assist braking assist rear brake diameter 130 overall width without mirrors 732 wheelbase 1169 overall length 1931 minimum ground clearance 56 overall height 576 rear track 641 front track 630 ege highway mpg 29 epa combined mpg 23 manufacturer 0 100km acceleration time seconds 58 aerodynamic drag cd 030 epa city mpg 19 turning diameter 392 curb weight 4233 tco curb weight 4233 epa highway mpg 29 fuel capacity 185 ege city mpg 19 height adjustable driver seat height adjustable driver seat adjustable lumbar power adjustable lumbar support driver seat whiplash protection whiplash protection system driver seat easy entry power steering wheel number of driver seat power adjustments 10 number of memorized driver seat settings 2 adjustable driver seat headrest power adjustable headrests 2nd row leg room 361 2nd row shoulder room 562 2nd row head room 383 1st row leg room 414 1st row head room 405 1st row shoulder room 583 epa interior volume 116 in dash cd single cd player audio system memory feature includes audio system digital audio input auxiliary audio input and ipodiphone integration total number of speakers 10 mp3 player cd mp3 playback antenna type diversity audio security system audio security system usb connection usb connection radio data system radio data system radio amfm hd radio watts 180 subwoofer 2 rear tire width 245 run flat tires run flat tire type all season rear tire diameter 18 rear tire profile 45 rear tire speed rating v rear tire width 245 run flat tires run flat tire type all season rear tire diameter 19 rear tire profile 40 in dash cd single cd player digital audio input auxiliary audio input and ipodiphone integration total number of speakers 16 mp3 player cd mp3 playback speed sensitive volume control speed sensitive volume control radio data system radio data system dvd audio dvdaudio radio amfm hd radio satellite radio sirius watts 600 surround audio 51 months of provided satellite radio service 12 rear tire width 245 run flat tires run flat tire type all season rear tire diameter 19 rear tire profile 40 warranty maximum mileage 50000 warranty miles limitedunlimited l warranty years limitedunlimited l warranty maximum years 4 warranty type basic warranty maximum mileage 50000 warranty miles limitedunlimited l warranty years limitedunlimited l warranty maximum years 4 warranty type drivetrain warranty maximum mileage 50000 warranty comments free full maintenance for 4 years or 50000 miles warranty miles limitedunlimited l warranty years limitedunlimited l warranty maximum years 4 warranty

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 67,441 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 14 tokens mean: 68.57 tokens max: 384 tokens	min: 15 tokens mean: 69.25 tokens max: 384 tokens

• Samples:

  anchor	positive
  barracuda essentials for office 365 complete protection and compliance li barracuda li email security cdwgcom five year license renewal for barracuda essentials for office 365 complete protection for up to 249 users otherelectronics	barracuda essentials for office 365 complete protection and compliance li barracuda li email security cdwcom five year license renewal for barracuda essentials for office 365 complete protection for up to 249 users otherelectronics
  cool carbon performance brake pad set rear 2003 bmw 325i base sedan upgrades page 2 automotive	cool carbon performance brake pad set rear 2002 bmw 325i base wagon upgrades page 2 automotive
  decoart americana acrylic paint williamsburg blue 2 ounces blue 2 oz cover art projects with affordable quality our 2ounce bottle of americana acrylic paint in williamsburg blue boasts a bright pigment that can be used on almost any surface once dry this paint will create a durable matte finish that can be varnished to create a variety of sheens blue acrylic paint permanent waterbased nontoxic lightfast pigment one coat coverage with most colors intermixable blends easily consistent viscosity made in usa toolsandhomeimprovement	decoart americana acrylic paint williamsburg blue 2 oz wholesale darice decoart americana acrylic paint williamsburg blue 2 oz decoart toolsandhomeimprovement

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• learning_rate: 1e-05
• num_train_epochs: 2
• warmup_ratio: 0.1
• fp16: True
• auto_find_batch_size: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: True
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss
0.2076	7000	0.015	0.0053
0.4152	14000	0.0059	0.0042
0.6228	21000	0.0041	0.0034
0.8303	28000	0.0038	0.0030
1.0379	35000	0.0028	0.0026
1.2455	42000	0.0023	0.0023
1.4531	49000	0.0011	0.0023
1.6607	56000	0.0008	0.0020
1.8683	63000	0.0007	0.0020

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.44.0
• PyTorch: 2.2.1
• Accelerate: 0.33.0
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, 
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}