spl4shedEdu/mpnet_ISM

SentenceTransformer based on sentence-transformers/all-mpnet-base-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-mpnet-base-v2. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-mpnet-base-v2
• Maximum Sequence Length: 384 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 384, 'do_lower_case': False}) with Transformer model: MPNetModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'paint sealant sonax profiline polymer net shield 75 ml aerosol can 1994 bmw 318is base coupe miscellaneous page 24 note innovative surface protection based on hybrid polymers protects the paintwork by means of a resistant network made from organic and inorganic components can be applied quickly easily intensively freshens up paint color produces silky smooth with an outstanding drip off effect one 75 ml should complete average size car produced by sonax identifiers is 223000m941 category of automotive',
    'paint sealant sonax profiline polymer net shield 75 ml aerosol can 1991 bmw 325i base convertible miscellaneous page 23 note innovative surface protection based on hybrid polymers protects the paintwork by means of a resistant network made from organic and inorganic components can be applied quickly easily intensively freshens up paint color produces silky smooth with an outstanding drip off effect one 75 ml should complete average size car produced by sonax identifiers is 223000m941 category of automotive',
    'honeywell accessories for terminal cod99exmb12 honeywell cod871238012 honeywell dolphin 99ex mobile base vehicle kit charging cradle rs232 universal mounting bracket and 12v cigarette lighter power adapter produced by honeywell metrologic identifiers is 99exmb12 category of computersandaccessories',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Training Details

Training Dataset

Unnamed Dataset

• Size: 281,342 training samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 28 tokens mean: 81.11 tokens max: 384 tokens	min: 28 tokens mean: 82.01 tokens max: 384 tokens

• Samples:

  anchor	positive
  honeywell hand held products dolphin 99509951 series mobile computer usb cable 6 ft 18m 80000355e usb cable 6 ft 18m identifiers is 80000355e category of computersandaccessories	hand held usb cable 6 ft hand ft 80000355e scanner accessories cdwcom hand held products is the leading provider of imagebased data collection solutions for mobile wireless and transaction processing applications to end users throughout world by investing in hhp products its customers are able reduce costs improve service position their companies future growth identifiers is 26121604 category of computersandaccessories
  intake boot air mass sensor to throttle housing 1995 bmw 318i base convertible intake system page 2 note from 0994 produced by oem identifiers is 13711247829m58 category of automotive	intake boot air mass sensor to throttle housing 1995 bmw 318i base convertible intake system page 2 produced by crp identifiers is 13711247829int category of automotive
  blue sky panorama with transparent clouds vector image sky images over 150 000 vector blue sky panorama with transparent clouds vector background image identifiers is 15266707 category of officeproducts	blue sky panorama with transparent clouds vector image images within landscapes nature over 55 000 vector blue sky panorama with transparent clouds vector background image identifiers is 15266707 category of officeproducts

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 70,336 evaluation samples
• Columns: anchor and positive
• Approximate statistics based on the first 1000 samples:

  	anchor	positive
  type	string	string
  details	min: 28 tokens mean: 84.74 tokens max: 384 tokens	min: 25 tokens mean: 87.69 tokens max: 384 tokens

• Samples:

  anchor	positive
  heater hose inlet from cylinder head to water valve 1997 bmw 318i base sedan heater system page 3 produced by genuine bmw identifiers is 64211394295boe category of automotive	heater hose inlet from cylinder head to water valve 1996 bmw 318i base convertible heater system page 3 produced by genuine bmw identifiers is 64211394295boe category of automotive
  harris harris group inc group 1 full quote netdaniacom produced by source nasdaq identifiers is isinus4138331040 category of toolsandhomeimprovement	harris harris group inc 1 statistics netdaniacom group produced by source nasdaq identifiers is isinus4138331040 category of toolsandhomeimprovement
  swiffer dusters with extendable handledusters plastic handle extends to 3 ft 1 per kit handledusters ft kitpag82074 buy online at janeice products identifiers is pag82074 category of toolsandhomeimprovement key specifications are weight per case std pkg quantity package one handle and three dusters description includes item cube 008276 upc code 037000447504 pack 00037000820741 length 092 width 022 height 042 0476	6 pack value bundle pag82074 dusters plastic handle extends to 3 ft 1 dusters per kitus feather page 5 the janitorial marketus now its easier than ever to get those hardtoreach places pivoting head can be adjusted and locked into place for cleaning angled surfaces such as ceiling fans cabinet corners baseboards refill dusters sold separately one handle three per box bristle material fiber color white plastic greenus produced by pag82074us identifiers is pag82074 category of toolsandhomeimprovement

• Loss: CachedMultipleNegativesRankingLoss with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• learning_rate: 1e-05
• num_train_epochs: 2
• warmup_ratio: 0.1
• fp16: True
• auto_find_batch_size: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 8
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: True
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• eval_use_gather_object: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	loss
0.1990	7000	0.0087	0.0027
0.3981	14000	0.0026	0.0020
0.5971	21000	0.0014	0.0018
0.7962	28000	0.0014	0.0014
0.9952	35000	0.0013	0.0010
1.1943	42000	0.0008	0.0010
1.3933	49000	0.0005	0.0010
1.5924	56000	0.0003	0.0009

Framework Versions

• Python: 3.10.13
• Sentence Transformers: 3.0.1
• Transformers: 4.44.0
• PyTorch: 2.2.1
• Accelerate: 0.33.0
• Datasets: 2.21.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup}, 
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}