sentence-transformers safetensors qwen2 sentence-similarity feature-extraction generated_from_trainer dataset_size:2859594 loss:MatryoshkaLoss loss:MultipleNegativesRankingLoss arxiv:1908.10084 arxiv:2205.13147 arxiv:1705.00652 base_model:Qwen/Qwen2.5-0.5B-Instruct base_model:finetune:Qwen/Qwen2.5-0.5B-Instruct model-index autotrain_compatible text-embeddings-inference endpoints_compatible region:us
---
base_model: Qwen/Qwen2.5-0.5B-Instruct
library_name: sentence-transformers
metrics:
- pearson_cosine
- spearman_cosine
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:2859594
- loss:MatryoshkaLoss
- loss:MultipleNegativesRankingLoss
widget:
- source_sentence: How old is Garry Marshall?
  sentences:
  - 'Garry Marshall

    On the morning of July 19, 2016, Marshall died at a hospital in Burbank, California
    at the age of 81 due to complications of pneumonia after suffering a stroke.[20][21]'
  - 'Gregg Marshall

    Michael Gregg Marshall (born February 27, 1963) is an American college basketball
    coach who currently leads the Shockers team at Wichita State University. Marshall
    has coached his teams to appearances in the NCAA Men''s Division I Basketball
    Tournament in twelve of his eighteen years as a head coach. He is the most successful
    head coach in Wichita State University history (261 wins), and is also the most
    successful head coach in Winthrop University history (194 wins).'
  - 'Guillotine

    For a period of time after its invention, the guillotine was called a louisette.
    However, it was later named after Guillotin who had proposed that a less painful
    method of execution should be found in place of the breaking wheel, though he
    opposed the death penalty and bemoaned the association of the guillotine with
    his name.'
- source_sentence: Are there cherry trees in Cherry Springs State Park?
  sentences:
  - 'Cherry Springs State Park

    Awards and press recognition have come to Cherry Springs and its staff. Thom Bemus,
    who initiated and coordinates the Stars-n-Parks program, was named DCNR''s 2002Volunteer
    of the Year.[66] In 2007the park''s Dark Sky Programming and staff received the
    Environmental Education Excellence in Programming award from the Pennsylvania
    Recreation and Parks Society.[67] Operations manager Chip Harrison and his wife
    Maxine, who directs the Dark Sky Fund, received a 2008award from the Pennsylvania
    Outdoor Lighting Council for "steadfast adherence and active promotion of the
    principles of responsible outdoor lighting at Cherry Springs State Park".[68]
    The DCNR has named Cherry Springs one of "25 Must-See Pennsylvania State Parks",
    specifically for having the "darkest night skies on the east coast".[69] Cherry
    Springs State Park was featured in the national press in 2003when USA Today named
    it one of "10Great Places to get some stars in your eyes",[70] in 2006when National
    Geographic Adventure featured it in "Pennsylvania: The Wild, Wild East",[71] and
    in The New York Times in 2007.[53] All these were before it was named an International
    Dark Sky Park by the International Dark-Sky Association in 2008.[38]'
  - 'Cantonese

    Although Cantonese shares a lot of vocabulary with Mandarin, the two varieties
    are mutually unintelligible because of differences in pronunciation, grammar and
    lexicon. Sentence structure, in particular the placement of verbs, sometimes differs
    between the two varieties. A notable difference between Cantonese and Mandarin
    is how the spoken word is written; both can be recorded verbatim, but very few
    Cantonese speakers are knowledgeable in the full Cantonese written vocabulary,
    so a non-verbatim formalized written form is adopted, which is more akin to the
    Mandarin written form.[4][5] This results in the situation in which a Cantonese
    and a Mandarin text may look similar but are pronounced differently.'
  - 'Cherry Springs State Park

    Cherry Springs State Park is an 82-acre (33ha)[a] Pennsylvania state park in Potter
    County, Pennsylvania, United States. The park was created from land within the
    Susquehannock State Forest, and is on Pennsylvania Route 44 in West Branch Township.
    Cherry Springs, named for a large stand of Black Cherry trees in the park, is
    atop the dissected Allegheny Plateau at an elevation of 2,300 feet (701m). It
    is popular with astronomers and stargazers for having "some of the darkest night
    skies on the east coast" of the United States, and was chosen by the Pennsylvania
    Department of Conservation and Natural Resources (DCNR) and its Bureau of Parks
    as one of "25 Must-See Pennsylvania State Parks".[4]'
- source_sentence: How many regions are in Belgium?
  sentences:
  - 'Pine City, Minnesota

    Pine City is a city in Pine County, Minnesota, in East Central Minnesota. Pine
    City is the county seat of, and the largest city in, Pine County.[7] A portion
    of the city is located on the Mille Lacs Indian Reservation. Founded as a railway
    town, it quickly became a logging community and the surrounding lakes made it
    a resort town. Today, it is an arts town and commuter town to jobs in the Minneapolis–Saint
    Paul metropolitan area.[8] It is also a green city.[9] The population was 3,127
    at the 2010 census.'
  - 'Provinces of Belgium

    The country of Belgium is divided into three regions. Two of these regions, the
    Flemish Region or Flanders, and Walloon Region, or Wallonia, are each subdivided
    into five provinces. The third region, the Brussels-Capital Region, is not divided
    into provinces, as it was originally only a small part of a province itself.'
  - 'United Belgian States

    The United Belgian States was a confederal republic of eight provinces which had
    their own governments, were sovereign and independent, and were governed directly
    by the Sovereign Congress (; ), the confederal government. The Sovereign Congress
    was seated in Brussels and consisted of representatives of each of the eight provinces.
    The provinces of the republic were divided into 11 smaller separate territories,
    each with their own regional identities:In 1789, a church-inspired popular revolt
    broke out in reaction to the emperor''s centralizing and anticlerical policies.
    Two factions appeared: the "Statists" who opposed the reforms, and the "Vonckists"
    named for Jan Frans Vonck who initially supported the reforms but then joined
    the opposition, due to the clumsy way in which the reforms were carried out.'
- source_sentence: Are there black holes near the galactic nucleus?
  sentences:
  - 'Supermassive black hole

    In September 2014, data from different X-ray telescopes has shown that the extremely
    small, dense, ultracompact dwarf galaxy M60-UCD1 hosts a 20 million solar mass
    black hole at its center, accounting for more than 10% of the total mass of the
    galaxy. The discovery is quite surprising, since the black hole is five times
    more massive than the Milky Way''s black hole despite the galaxy being less than
    five-thousandths the mass of the Milky Way.'
  - 'Aquarela do Brasil

    "Aquarela do Brasil" (Portuguese:[akwaˈɾɛlɐ du bɾaˈziw], Watercolor of Brazil),
    written by Ary Barroso in 1939 and known in the English-speaking world simply
    as "Brazil", is one of the most famous Brazilian songs.'
  - 'Supermassive black hole

    The difficulty in forming a supermassive black hole resides in the need for enough
    matter to be in a small enough volume. This matter needs to have very little angular
    momentum in order for this to happen. Normally, the process of accretion involves
    transporting a large initial endowment of angular momentum outwards, and this
    appears to be the limiting factor in black hole growth. This is a major component
    of the theory of accretion disks. Gas accretion is the most efficient and also
    the most conspicuous way in which black holes grow. The majority of the mass growth
    of supermassive black holes is thought to occur through episodes of rapid gas
    accretion, which are observable as active galactic nuclei or quasars. Observations
    reveal that quasars were much more frequent when the Universe was younger, indicating
    that supermassive black holes formed and grew early. A major constraining factor
    for theories of supermassive black hole formation is the observation of distant
    luminous quasars, which indicate that supermassive black holes of billions of
    solar masses had already formed when the Universe was less than one billion years
    old. This suggests that supermassive black holes arose very early in the Universe,
    inside the first massive galaxies.'
- source_sentence: When did the July Monarchy end?
  sentences:
  - 'July Monarchy

    Despite the return of the House of Bourbon to power, France was much changed from
    the era of the ancien régime. The egalitarianism and liberalism of the revolutionaries
    remained an important force and the autocracy and hierarchy of the earlier era
    could not be fully restored. Economic changes, which had been underway long before
    the revolution, had progressed further during the years of turmoil and were firmly
    entrenched by 1815. These changes had seen power shift from the noble landowners
    to the urban merchants. The administrative reforms of Napoleon, such as the Napoleonic
    Code and efficient bureaucracy, also remained in place. These changes produced
    a unified central government that was fiscally sound and had much control over
    all areas of French life, a sharp difference from the complicated mix of feudal
    and absolutist traditions and institutions of pre-Revolutionary Bourbons.'
  - 'Wachovia

    Wachovia Corporation began on June 16, 1879 in Winston-Salem, North Carolina as
    the Wachovia National Bank. The bank was co-founded by James Alexander Gray and
    William Lemly.[9] In 1911, the bank merged with Wachovia Loan and Trust Company,
    "the largest trust company between Baltimore and New Orleans",[10] which had been
    founded on June 15, 1893. Wachovia grew to become one of the largest banks in
    the Southeast partly on the strength of its accounts from the R.J. Reynolds Tobacco
    Company, which was also headquartered in Winston-Salem.[11] On December 12, 1986,
    Wachovia purchased First Atlanta. Founded as Atlanta National Bank on September
    14, 1865, and later renamed to First National Bank of Atlanta, this institution
    was the oldest national bank in Atlanta. This purchase made Wachovia one of the
    few companies with dual headquarters: one in Winston-Salem and one in Atlanta.
    In 1991, Wachovia entered the South Carolina market by acquiring South Carolina
    National Corporation,[12] founded as the Bank of Charleston in 1834. In 1998,
    Wachovia acquired two Virginia-based banks, Jefferson National Bank and Central
    Fidelity Bank. In 1997, Wachovia acquired both 1st United Bancorp and American
    Bankshares Inc, giving its first entry into Florida. In 2000, Wachovia made its
    final purchase, which was Republic Security Bank.'
  - 'July Monarchy

    The July Monarchy (French: Monarchie de Juillet) was a liberal constitutional
    monarchy in France under Louis Philippe I, starting with the July Revolution of
    1830 and ending with the Revolution of 1848. It marks the end of the Bourbon Restoration
    (1814–1830). It began with the overthrow of the conservative government of Charles
    X, the last king of the House of Bourbon.'
model-index:
- name: SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct
  results:
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 896
      type: sts-dev-896
    metrics:
    - type: pearson_cosine
      value: 0.45729692013517886
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.49645340246652353
      name: Spearman Cosine
  - task:
      type: semantic-similarity
      name: Semantic Similarity
    dataset:
      name: sts dev 768
      type: sts-dev-768
    metrics:
    - type: pearson_cosine
      value: 0.4455125981991164
      name: Pearson Cosine
    - type: spearman_cosine
      value: 0.4896539219726307
      name: Spearman Cosine
---

# SentenceTransformer based on Qwen/Qwen2.5-0.5B-Instruct

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct). It maps sentences & paragraphs to a 896-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct) <!-- at revision 7ae557604adf67be50417f59c2c2f167def9a775 -->
- **Maximum Sequence Length:** 1024 tokens
- **Output Dimensionality:** 896 dimensions
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 1024, 'do_lower_case': False}) with Transformer model: Qwen2Model 
  (1): Pooling({'word_embedding_dimension': 896, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("AlexWortega/qwen1k")
# Run inference
sentences = [
    'When did the July Monarchy end?',
    'July Monarchy\nThe July Monarchy (French: Monarchie de Juillet) was a liberal constitutional monarchy in France under Louis Philippe I, starting with the July Revolution of 1830 and ending with the Revolution of 1848. It marks the end of the Bourbon Restoration (1814–1830). It began with the overthrow of the conservative government of Charles X, the last king of the House of Bourbon.',
    'July Monarchy\nDespite the return of the House of Bourbon to power, France was much changed from the era of the ancien régime. The egalitarianism and liberalism of the revolutionaries remained an important force and the autocracy and hierarchy of the earlier era could not be fully restored. Economic changes, which had been underway long before the revolution, had progressed further during the years of turmoil and were firmly entrenched by 1815. These changes had seen power shift from the noble landowners to the urban merchants. The administrative reforms of Napoleon, such as the Napoleonic Code and efficient bureaucracy, also remained in place. These changes produced a unified central government that was fiscally sound and had much control over all areas of French life, a sharp difference from the complicated mix of feudal and absolutist traditions and institutions of pre-Revolutionary Bourbons.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 896]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Semantic Similarity

* Datasets: `sts-dev-896` and `sts-dev-768`
* Evaluated with [<code>EmbeddingSimilarityEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.EmbeddingSimilarityEvaluator)

| Metric              | sts-dev-896 | sts-dev-768 |
|:--------------------|:------------|:------------|
| pearson_cosine      | 0.4573      | 0.4455      |
| **spearman_cosine** | **0.4965**  | **0.4897**  |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Dataset

#### Unnamed Dataset


* Size: 2,859,594 training samples
* Columns: <code>query</code>, <code>response</code>, and <code>negative</code>
* Approximate statistics based on the first 1000 samples:
  |         | query                                                                            | response                                                                             | negative                                                                            |
  |:--------|:---------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------|
  | type    | string                                                                           | string                                                                               | string                                                                              |
  | details | <ul><li>min: 4 tokens</li><li>mean: 8.76 tokens</li><li>max: 26 tokens</li></ul> | <ul><li>min: 23 tokens</li><li>mean: 141.88 tokens</li><li>max: 532 tokens</li></ul> | <ul><li>min: 4 tokens</li><li>mean: 134.02 tokens</li><li>max: 472 tokens</li></ul> |
* Samples:
  | query                                             | response                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 | negative                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
  |:--------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
  | <code>Was there a year 0?</code>                  | <code>Year zero<br>Year zero does not exist in the anno Domini system usually used to number years in the Gregorian calendar and in its predecessor, the Julian calendar. In this system, the year 1 BC is followed by AD 1. However, there is a year zero in astronomical year numbering (where it coincides with the Julian year 1 BC) and in ISO 8601:2004 (where it coincides with the Gregorian year 1 BC) as well as in all Buddhist and Hindu calendars.</code>                                                                                                                                                                                                                                                                                                                                   | <code>504<br>Year 504 (DIV) was a leap year starting on Thursday (link will display the full calendar) of the Julian calendar. At the time, it was known as the Year of the Consulship of Nicomachus without colleague (or, less frequently, year 1257 "Ab urbe condita"). The denomination 504 for this year has been used since the early medieval period, when the Anno Domini calendar era became the prevalent method in Europe for naming years.</code>                                                                                                                                                                                                                                                                                             |
  | <code>When is the dialectical method used?</code> | <code>Dialectic<br>Dialectic or dialectics (Greek: διαλεκτική, dialektikḗ; related to dialogue), also known as the dialectical method, is at base a discourse between two or more people holding different points of view about a subject but wishing to establish the truth through reasoned arguments. Dialectic resembles debate, but the concept excludes subjective elements such as emotional appeal and the modern pejorative sense of rhetoric.[1][2] Dialectic may be contrasted with the didactic method, wherein one side of the conversation teaches the other. Dialectic is alternatively known as minor logic, as opposed to major logic or critique.</code>                                                                                                                               | <code>Derek Bentley case<br>Another factor in the posthumous defence was that a "confession" recorded by Bentley, which was claimed by the prosecution to be a "verbatim record of dictated monologue", was shown by forensic linguistics methods to have been largely edited by policemen. Linguist Malcolm Coulthard showed that certain patterns, such as the frequency of the word "then" and the grammatical use of "then" after the grammatical subject ("I then" rather than "then I"), were not consistent with Bentley's use of language (his idiolect), as evidenced in court testimony. These patterns fit better the recorded testimony of the policemen involved. This is one of the earliest uses of forensic linguistics on record.</code> |
  | <code>What do Grasshoppers eat?</code>            | <code>Grasshopper<br>Grasshoppers are plant-eaters, with a few species at times becoming serious pests of cereals, vegetables and pasture, especially when they swarm in their millions as locusts and destroy crops over wide areas. They protect themselves from predators by camouflage; when detected, many species attempt to startle the predator with a brilliantly-coloured wing-flash while jumping and (if adult) launching themselves into the air, usually flying for only a short distance. Other species such as the rainbow grasshopper have warning coloration which deters predators. Grasshoppers are affected by parasites and various diseases, and many predatory creatures feed on both nymphs and adults. The eggs are the subject of attack by parasitoids and predators.</code> | <code>Groundhog<br>Very often the dens of groundhogs provide homes for other animals including skunks, red foxes, and cottontail rabbits. The fox and skunk feed upon field mice, grasshoppers, beetles and other creatures that destroy farm crops. In aiding these animals, the groundhog indirectly helps the farmer. In addition to providing homes for itself and other animals, the groundhog aids in soil improvement by bringing subsoil to the surface. The groundhog is also a valuable game animal and is considered a difficult sport when hunted in a fair manner. In some parts of Appalachia, they are eaten.</code>                                                                                                                       |
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters:
  ```json
  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          896,
          768
      ],
      "matryoshka_weights": [
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  ```

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: steps
- `per_device_train_batch_size`: 12
- `per_device_eval_batch_size`: 12
- `gradient_accumulation_steps`: 4
- `num_train_epochs`: 1
- `warmup_ratio`: 0.3
- `bf16`: True
- `batch_sampler`: no_duplicates

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 12
- `per_device_eval_batch_size`: 12
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 4
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 5e-05
- `weight_decay`: 0.0
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 1
- `max_steps`: -1
- `lr_scheduler_type`: linear
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0.3
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: False
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: None
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: None
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `include_for_metrics`: []
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `average_tokens_across_devices`: False
- `prompts`: None
- `batch_sampler`: no_duplicates
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
| Epoch  | Step | Training Loss | sts-dev-896_spearman_cosine | sts-dev-768_spearman_cosine |
|:------:|:----:|:-------------:|:---------------------------:|:---------------------------:|
| 0.0002 | 10   | 4.4351        | -                           | -                           |
| 0.0003 | 20   | 4.6508        | -                           | -                           |
| 0.0005 | 30   | 4.7455        | -                           | -                           |
| 0.0007 | 40   | 4.5427        | -                           | -                           |
| 0.0008 | 50   | 4.3982        | -                           | -                           |
| 0.0010 | 60   | 4.3755        | -                           | -                           |
| 0.0012 | 70   | 4.4105        | -                           | -                           |
| 0.0013 | 80   | 5.2227        | -                           | -                           |
| 0.0015 | 90   | 5.8062        | -                           | -                           |
| 0.0017 | 100  | 5.7645        | -                           | -                           |
| 0.0018 | 110  | 5.9261        | -                           | -                           |
| 0.0020 | 120  | 5.8301        | -                           | -                           |
| 0.0022 | 130  | 5.7602        | -                           | -                           |
| 0.0023 | 140  | 5.9392        | -                           | -                           |
| 0.0025 | 150  | 5.7523        | -                           | -                           |
| 0.0027 | 160  | 5.8585        | -                           | -                           |
| 0.0029 | 170  | 5.7916        | -                           | -                           |
| 0.0030 | 180  | 5.8157        | -                           | -                           |
| 0.0032 | 190  | 5.7102        | -                           | -                           |
| 0.0034 | 200  | 5.5844        | -                           | -                           |
| 0.0035 | 210  | 5.5463        | -                           | -                           |
| 0.0037 | 220  | 5.5823        | -                           | -                           |
| 0.0039 | 230  | 5.5514        | -                           | -                           |
| 0.0040 | 240  | 5.5646        | -                           | -                           |
| 0.0042 | 250  | 5.5783        | -                           | -                           |
| 0.0044 | 260  | 5.5344        | -                           | -                           |
| 0.0045 | 270  | 5.523         | -                           | -                           |
| 0.0047 | 280  | 5.4969        | -                           | -                           |
| 0.0049 | 290  | 5.5407        | -                           | -                           |
| 0.0050 | 300  | 5.6171        | -                           | -                           |
| 0.0052 | 310  | 5.5581        | -                           | -                           |
| 0.0054 | 320  | 5.8903        | -                           | -                           |
| 0.0055 | 330  | 5.8675        | -                           | -                           |
| 0.0057 | 340  | 5.745         | -                           | -                           |
| 0.0059 | 350  | 5.6041        | -                           | -                           |
| 0.0060 | 360  | 5.5476        | -                           | -                           |
| 0.0062 | 370  | 5.3964        | -                           | -                           |
| 0.0064 | 380  | 5.3564        | -                           | -                           |
| 0.0065 | 390  | 5.3054        | -                           | -                           |
| 0.0067 | 400  | 5.2779        | -                           | -                           |
| 0.0069 | 410  | 5.206         | -                           | -                           |
| 0.0070 | 420  | 5.2168        | -                           | -                           |
| 0.0072 | 430  | 5.1645        | -                           | -                           |
| 0.0074 | 440  | 5.1797        | -                           | -                           |
| 0.0076 | 450  | 5.2526        | -                           | -                           |
| 0.0077 | 460  | 5.1768        | -                           | -                           |
| 0.0079 | 470  | 5.3519        | -                           | -                           |
| 0.0081 | 480  | 5.2982        | -                           | -                           |
| 0.0082 | 490  | 5.3229        | -                           | -                           |
| 0.0084 | 500  | 5.3758        | -                           | -                           |
| 0.0086 | 510  | 5.2478        | -                           | -                           |
| 0.0087 | 520  | 5.1799        | -                           | -                           |
| 0.0089 | 530  | 5.1088        | -                           | -                           |
| 0.0091 | 540  | 4.977         | -                           | -                           |
| 0.0092 | 550  | 4.9108        | -                           | -                           |
| 0.0094 | 560  | 4.811         | -                           | -                           |
| 0.0096 | 570  | 4.7203        | -                           | -                           |
| 0.0097 | 580  | 4.6499        | -                           | -                           |
| 0.0099 | 590  | 4.4548        | -                           | -                           |
| 0.0101 | 600  | 4.2891        | -                           | -                           |
| 0.0102 | 610  | 4.1881        | -                           | -                           |
| 0.0104 | 620  | 4.6           | -                           | -                           |
| 0.0106 | 630  | 4.5365        | -                           | -                           |
| 0.0107 | 640  | 4.3086        | -                           | -                           |
| 0.0109 | 650  | 4.0452        | -                           | -                           |
| 0.0111 | 660  | 3.9041        | -                           | -                           |
| 0.0112 | 670  | 4.3938        | -                           | -                           |
| 0.0114 | 680  | 4.3198        | -                           | -                           |
| 0.0116 | 690  | 4.1294        | -                           | -                           |
| 0.0117 | 700  | 4.077         | -                           | -                           |
| 0.0119 | 710  | 3.9174        | -                           | -                           |
| 0.0121 | 720  | 4.1629        | -                           | -                           |
| 0.0123 | 730  | 3.9611        | -                           | -                           |
| 0.0124 | 740  | 3.7768        | -                           | -                           |
| 0.0126 | 750  | 3.5842        | -                           | -                           |
| 0.0128 | 760  | 3.1196        | -                           | -                           |
| 0.0129 | 770  | 3.6288        | -                           | -                           |
| 0.0131 | 780  | 3.273         | -                           | -                           |
| 0.0133 | 790  | 2.7889        | -                           | -                           |
| 0.0134 | 800  | 2.5096        | -                           | -                           |
| 0.0136 | 810  | 1.8878        | -                           | -                           |
| 0.0138 | 820  | 2.3423        | -                           | -                           |
| 0.0139 | 830  | 1.7687        | -                           | -                           |
| 0.0141 | 840  | 2.0781        | -                           | -                           |
| 0.0143 | 850  | 2.4598        | -                           | -                           |
| 0.0144 | 860  | 1.7667        | -                           | -                           |
| 0.0146 | 870  | 2.6247        | -                           | -                           |
| 0.0148 | 880  | 1.916         | -                           | -                           |
| 0.0149 | 890  | 2.0817        | -                           | -                           |
| 0.0151 | 900  | 2.3679        | -                           | -                           |
| 0.0153 | 910  | 1.418         | -                           | -                           |
| 0.0154 | 920  | 2.7353        | -                           | -                           |
| 0.0156 | 930  | 1.992         | -                           | -                           |
| 0.0158 | 940  | 1.4564        | -                           | -                           |
| 0.0159 | 950  | 1.4154        | -                           | -                           |
| 0.0161 | 960  | 0.9499        | -                           | -                           |
| 0.0163 | 970  | 1.6304        | -                           | -                           |
| 0.0164 | 980  | 0.9264        | -                           | -                           |
| 0.0166 | 990  | 1.3278        | -                           | -                           |
| 0.0168 | 1000 | 1.686         | 0.4965                      | 0.4897                      |


### Framework Versions
- Python: 3.10.12
- Sentence Transformers: 3.3.0
- Transformers: 4.46.2
- PyTorch: 2.1.0+cu118
- Accelerate: 1.1.1
- Datasets: 3.1.0
- Tokenizers: 0.20.3

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### MatryoshkaLoss
```bibtex
@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning},
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
```

#### MultipleNegativesRankingLoss
```bibtex
@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->