sentence-transformers safetensors modernbert sentence-similarity feature-extraction generated_from_trainer dataset_size:3012496 loss:CachedMultipleNegativesRankingLoss en dataset:sentence-transformers/gooaq arxiv:1908.10084 arxiv:2101.06983 base_model:nomic-ai/modernbert-embed-base base_model:finetune:nomic-ai/modernbert-embed-base autotrain_compatible endpoints_compatible region:us
---
base_model: nomic-ai/modernbert-embed-base
datasets:
- sentence-transformers/gooaq
language:
- en
library_name: sentence-transformers
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:3012496
- loss:CachedMultipleNegativesRankingLoss
widget:
- source_sentence: how long does it take to cook a 3 pound ham?
  sentences:
  - Preheat the oven to 325°F. Place the ham on a rack in a shallow roasting pan.
    For a whole 10- to 15-pound ham, allow 18 to 20 minutes per pound; for a half--5
    to 7 pounds--about 20 minutes per pound; or for a shank or butt portion weighing
    3 to 4 pounds, about 35 minutes to the pound.
  - The endoscope doesn't interfere with your breathing, most patients consider the
    test only slightly uncomfortable, and many patients fall asleep during the procedure.
    This procedure can also be called an upper GI endoscopy, Esophagogastroduodenoscopy
    (EGD) or pan endoscopy.
  - 'The third letter in the personality type acronym corresponds to the preference
    within the thinking-feeling pair: “T” stands for thinking and “F” stands for feeling.
    ... ISTJ stands for Introverted, Sensing, Thinking, Judging. ENFP stands for Extraverted,
    iNtuitive, Feeling, Perceiving.'
- source_sentence: which aldi slim well meals are syn free?
  sentences:
  - Milia are tiny bumps that occur under the outer skin layer of the eyelid, around
    the eyes and nose, and on the chin or cheeks. Sometimes called "milk spots" or
    "oil seeds," these pearly white or yellowish cysts often appear in clusters and
    may be on large areas of the face. Milia occur most commonly in babies.
  - The full range includes Slim Free Moroccan Vegetable Stew, Slim Free Three Bean
    and Vegetable Chilli, Slim Free Chicken Saag, Slim Free Tikka Masala and Slim
    Free Meatballs and Pasta.
  - If your message is not delivered yet, that means the problem is on the recipient
    side. It could be a server problem, internet problem, settings problem, or anything
    else. ... Your friend or recipient has deliberately ignored your message. The
    recipient might have read your message from the notification or status bar.
- source_sentence: do redguards have last names?
  sentences:
  - While some hot peppers may not be toxic for dogs, dogs are not used to eating
    spicy foods so they are likely to experience some digestive upset after eating
    hot peppers. Bread and butter pickles are dangerous because they often contain
    onions and garlic pickles are bad because they are made with garlic.
  - All Redguard names are unisex and they have no surnames.
  - Abstract. White spots were observed on the mucosa immediately adjacent to polyps
    and carcinomas; the majority of the polyps proved to be carcinoma in situ or had
    invasive carcinoma. The white spots consisted of accumulations of foamy cells
    with features similar to muciphage.
- source_sentence: are queen and full the same size?
  sentences:
  - Queen mattress dimensions are 60 inches wide by approximately 80 inches long –
    7 inches wider and 5 inches longer than a full-size mattress. These added inches
    can make all the difference in comfort, especially for couples, and have made
    the queen-size mattress today's most popular mattress size.
  - When prepared on whole wheat bread, a PB&J sandwich made with two Tbsps. of peanut
    butter and two Tbsps. of grape jelly adds up to a whopping 530 calories, 460 mg
    of sodium, 74 grams of carbs, 35 grams sugar and 20 grams of fat.
  - Ryanair has cancelled 22 flights on Wednesday evening and 72 flights on Thursday
    as a result of the 14th French ATC strike, with further delays likely. ... Aer
    Lingus flights are also affected and here they are.
- source_sentence: is bmw 225xe 4 wheel drive?
  sentences:
  - The BMW 225xe offers both a higher system output and more boot capacity than its
    competitors. With its plug-in hybrid drive system, the BMW 225xe combines BMW
    EfficientDynamics with comfort, driving pleasure and all-wheel drive, and brings
    versatility and generous levels of space together in a compact vehicle.
  - The newest AP Top 25 Poll has Kentucky checking in at No. 13. Baylor, Gonzaga,
    Kansas, San Diego State and Florida State make up the top five. The Wildcats stand
    pat at No.
  - Tap Settings > [your name] > Password & Security. Tap Change Password. Enter your
    current password or device passcode, then enter a new password and confirm the
    new password. Tap Change or Change Password.
---

# SentenceTransformer based on nomic-ai/modernbert-embed-base

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [nomic-ai/modernbert-embed-base](https://huggingface.co/nomic-ai/modernbert-embed-base) on the [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [nomic-ai/modernbert-embed-base](https://huggingface.co/nomic-ai/modernbert-embed-base) <!-- at revision 5960f1566fb7cb1adf1eb6e816639cf4646d9b12 -->
- **Maximum Sequence Length:** 8192 tokens
- **Output Dimensionality:** 768 dimensions
- **Similarity Function:** Cosine Similarity
- **Training Dataset:**
    - [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq)
- **Language:** en
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False}) with Transformer model: ModernBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("Areeb-02/modernbert-embed-base-gooaq-8e-05")
# Run inference
sentences = [
    'is bmw 225xe 4 wheel drive?',
    'The BMW 225xe offers both a higher system output and more boot capacity than its competitors. With its plug-in hybrid drive system, the BMW 225xe combines BMW EfficientDynamics with comfort, driving pleasure and all-wheel drive, and brings versatility and generous levels of space together in a compact vehicle.',
    'Tap Settings > [your name] > Password & Security. Tap Change Password. Enter your current password or device passcode, then enter a new password and confirm the new password. Tap Change or Change Password.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### gooaq

* Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c)
* Size: 3,012,496 training samples
* Columns: <code>question</code> and <code>answer</code>
* Approximate statistics based on the first 1000 samples:
  |         | question                                                                         | answer                                                                              |
  |:--------|:---------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
  | type    | string                                                                           | string                                                                              |
  | details | <ul><li>min: 8 tokens</li><li>mean: 12.0 tokens</li><li>max: 21 tokens</li></ul> | <ul><li>min: 15 tokens</li><li>mean: 58.17 tokens</li><li>max: 190 tokens</li></ul> |
* Samples:
  | question                                                                     | answer                                                                                                                                                                                                                                                                                                                           |
  |:-----------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  | <code>what is the difference between clay and mud mask?</code>               | <code>The main difference between the two is that mud is a skin-healing agent, while clay is a cosmetic, drying agent. Clay masks are most useful for someone who has oily skin and is prone to breakouts of acne and blemishes.</code>                                                                                          |
  | <code>myki how much on card?</code>                                          | <code>A full fare myki card costs $6 and a concession, seniors or child myki costs $3. For more information about how to use your myki, visit ptv.vic.gov.au or call 1800 800 007.</code>                                                                                                                                        |
  | <code>how to find out if someone blocked your phone number on iphone?</code> | <code>If you get a notification like "Message Not Delivered" or you get no notification at all, that's a sign of a potential block. Next, you could try calling the person. If the call goes right to voicemail or rings once (or a half ring) then goes to voicemail, that's further evidence you may have been blocked.</code> |
* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
  ```json
  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  ```

### Evaluation Dataset

#### gooaq

* Dataset: [gooaq](https://huggingface.co/datasets/sentence-transformers/gooaq) at [b089f72](https://huggingface.co/datasets/sentence-transformers/gooaq/tree/b089f728748a068b7bc5234e5bcf5b25e3c8279c)
* Size: 3,012,496 evaluation samples
* Columns: <code>question</code> and <code>answer</code>
* Approximate statistics based on the first 1000 samples:
  |         | question                                                                          | answer                                                                              |
  |:--------|:----------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
  | type    | string                                                                            | string                                                                              |
  | details | <ul><li>min: 8 tokens</li><li>mean: 11.88 tokens</li><li>max: 18 tokens</li></ul> | <ul><li>min: 17 tokens</li><li>mean: 58.24 tokens</li><li>max: 110 tokens</li></ul> |
* Samples:
  | question                                                                      | answer                                                                                                                                                                                                                                                                                                      |
  |:------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  | <code>what are the four most common types of shopping centers quizlet?</code> | <code>['the neighborhood center.', 'the community center.', 'the regional center.', 'the super-regional center.']</code>                                                                                                                                                                                    |
  | <code>how far back does a enhanced dbs check go?</code>                       | <code>The filtering periods for cautions are two years for under 18s and six years for those aged 18 and over. The filtering periods for convictions are 5.5 years for under 18s and 11 years for those aged 18 and over.</code>                                                                            |
  | <code>can ezpass be used in colorado?</code>                                  | <code>ExpressToll transponder, switchable transponder, EZPass. ... ExpressToll passes only work in the State of Colorado. Travelers from out-of-state can use the express lanes and are billed via License Plate Toll. Formerly there was a separate tolling system for users of E470 called EZPass.</code> |
* Loss: [<code>CachedMultipleNegativesRankingLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#cachedmultiplenegativesrankingloss) with these parameters:
  ```json
  {
      "scale": 20.0,
      "similarity_fct": "cos_sim"
  }
  ```

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: steps
- `per_device_train_batch_size`: 30
- `per_device_eval_batch_size`: 30
- `learning_rate`: 8e-05
- `num_train_epochs`: 2
- `warmup_ratio`: 0.05
- `fp16`: True
- `batch_sampler`: no_duplicates

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 30
- `per_device_eval_batch_size`: 30
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 8e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 2
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.05
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: False
- `fp16`: True
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: None
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.1471 | 5    | 0.0452        | -               |
| 0.2941 | 10   | 0.0202        | -               |
| 0.4412 | 15   | 0.0227        | -               |
| 0.5882 | 20   | 0.0258        | -               |
| 0.7353 | 25   | 0.0361        | -               |
| 0.8824 | 30   | 0.03          | -               |
| 1.0294 | 35   | 0.0246        | -               |
| 1.1765 | 40   | 0.0036        | -               |
| 1.3235 | 45   | 0.0019        | -               |
| 1.4706 | 50   | 0.0021        | 0.0161          |
| 1.6176 | 55   | 0.0057        | -               |
| 1.7647 | 60   | 0.0083        | -               |
| 1.9118 | 65   | 0.0024        | -               |


### Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.3.1
- Transformers: 4.48.0.dev0
- PyTorch: 2.5.1+cu121
- Accelerate: 1.2.1
- Datasets: 3.2.0
- Tokenizers: 0.21.0

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### CachedMultipleNegativesRankingLoss
```bibtex
@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->