sentence-transformers safetensors bert sentence-similarity feature-extraction generated_from_trainer dataset_size:9504 loss:TripletLoss arxiv:1908.10084 arxiv:1703.07737 base_model:dbourget/philai-embeddings-2.0 base_model:finetune:dbourget/philai-embeddings-2.0 model-index autotrain_compatible text-embeddings-inference endpoints_compatible region:us
---
base_model: dbourget/philai-embeddings-2.0
library_name: sentence-transformers
metrics:
- cosine_accuracy
- dot_accuracy
- manhattan_accuracy
- euclidean_accuracy
- max_accuracy
pipeline_tag: sentence-similarity
tags:
- sentence-transformers
- sentence-similarity
- feature-extraction
- generated_from_trainer
- dataset_size:9504
- loss:TripletLoss
widget:
- source_sentence: cap product
  sentences:
  - method of adjoining a chain of degree p with a co-chain of degree q, where q is
    less than or equal to p, to form a composite chain of degree p-q
  - 'Ontology '
  - hat commodity
- source_sentence: cognitivism
  sentences:
  - supporting cognitive science
  - study of changes in organisms caused by modification of gene expression rather
    than alteration of the genetic code
  - 'the idea that mind works like an algorithmic symbol manipulation '
- source_sentence: doxastic voluntarism
  sentences:
  - Land surrounded by water
  - belief one is free
  - the ability to will beliefs
- source_sentence: conceptual role
  sentences:
  - concept
  - inferential role
  - 'Theory of knowledge '
- source_sentence: scientific revolutions
  sentences:
  - scientific realism
  - Universal moral principles govern legal systems
  - paradigm shifts
model-index:
- name: SentenceTransformer based on dbourget/philai-embeddings-2.0
  results:
  - task:
      type: triplet
      name: Triplet
    dataset:
      name: beatai dev
      type: beatai-dev
    metrics:
    - type: cosine_accuracy
      value: 0.8215488215488216
      name: Cosine Accuracy
    - type: dot_accuracy
      value: 0.24494949494949494
      name: Dot Accuracy
    - type: manhattan_accuracy
      value: 0.835016835016835
      name: Manhattan Accuracy
    - type: euclidean_accuracy
      value: 0.8341750841750841
      name: Euclidean Accuracy
    - type: max_accuracy
      value: 0.835016835016835
      name: Max Accuracy
---

# SentenceTransformer based on dbourget/philai-embeddings-2.0

This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [dbourget/philai-embeddings-2.0](https://huggingface.co/dbourget/philai-embeddings-2.0). It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

## Model Details

### Model Description
- **Model Type:** Sentence Transformer
- **Base model:** [dbourget/philai-embeddings-2.0](https://huggingface.co/dbourget/philai-embeddings-2.0) <!-- at revision d9add3b37c9bea5883418ac3f1d45cb29fe3a1dc -->
- **Maximum Sequence Length:** 512 tokens
- **Output Dimensionality:** 1024 tokens
- **Similarity Function:** Cosine Similarity
<!-- - **Training Dataset:** Unknown -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers)
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers)

### Full Model Architecture

```
SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
```

## Usage

### Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

```bash
pip install -U sentence-transformers
```

Then you can load this model and run inference.
```python
from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("dbourget/pb-small-10e-tsdae6e-philsim-cosine-6e-beatai-30e")
# Run inference
sentences = [
    'scientific revolutions',
    'paradigm shifts',
    'scientific realism',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
```

<!--
### Direct Usage (Transformers)

<details><summary>Click to see the direct usage in Transformers</summary>

</details>
-->

<!--
### Downstream Usage (Sentence Transformers)

You can finetune this model on your own dataset.

<details><summary>Click to expand</summary>

</details>
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

## Evaluation

### Metrics

#### Triplet
* Dataset: `beatai-dev`
* Evaluated with [<code>TripletEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.TripletEvaluator)

| Metric             | Value     |
|:-------------------|:----------|
| cosine_accuracy    | 0.8215    |
| dot_accuracy       | 0.2449    |
| manhattan_accuracy | 0.835     |
| euclidean_accuracy | 0.8342    |
| **max_accuracy**   | **0.835** |

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Hyperparameters
#### Non-Default Hyperparameters

- `eval_strategy`: steps
- `per_device_train_batch_size`: 138
- `per_device_eval_batch_size`: 138
- `learning_rate`: 1e-06
- `weight_decay`: 0.01
- `num_train_epochs`: 20
- `lr_scheduler_type`: constant
- `bf16`: True
- `dataloader_drop_last`: True
- `resume_from_checkpoint`: True

#### All Hyperparameters
<details><summary>Click to expand</summary>

- `overwrite_output_dir`: False
- `do_predict`: False
- `eval_strategy`: steps
- `prediction_loss_only`: True
- `per_device_train_batch_size`: 138
- `per_device_eval_batch_size`: 138
- `per_gpu_train_batch_size`: None
- `per_gpu_eval_batch_size`: None
- `gradient_accumulation_steps`: 1
- `eval_accumulation_steps`: None
- `torch_empty_cache_steps`: None
- `learning_rate`: 1e-06
- `weight_decay`: 0.01
- `adam_beta1`: 0.9
- `adam_beta2`: 0.999
- `adam_epsilon`: 1e-08
- `max_grad_norm`: 1.0
- `num_train_epochs`: 20
- `max_steps`: -1
- `lr_scheduler_type`: constant
- `lr_scheduler_kwargs`: {}
- `warmup_ratio`: 0
- `warmup_steps`: 0
- `log_level`: passive
- `log_level_replica`: warning
- `log_on_each_node`: True
- `logging_nan_inf_filter`: True
- `save_safetensors`: True
- `save_on_each_node`: False
- `save_only_model`: False
- `restore_callback_states_from_checkpoint`: False
- `no_cuda`: False
- `use_cpu`: False
- `use_mps_device`: False
- `seed`: 42
- `data_seed`: None
- `jit_mode_eval`: False
- `use_ipex`: False
- `bf16`: True
- `fp16`: False
- `fp16_opt_level`: O1
- `half_precision_backend`: auto
- `bf16_full_eval`: False
- `fp16_full_eval`: False
- `tf32`: None
- `local_rank`: 0
- `ddp_backend`: None
- `tpu_num_cores`: None
- `tpu_metrics_debug`: False
- `debug`: []
- `dataloader_drop_last`: True
- `dataloader_num_workers`: 0
- `dataloader_prefetch_factor`: 2
- `past_index`: -1
- `disable_tqdm`: False
- `remove_unused_columns`: True
- `label_names`: None
- `load_best_model_at_end`: False
- `ignore_data_skip`: False
- `fsdp`: []
- `fsdp_min_num_params`: 0
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
- `fsdp_transformer_layer_cls_to_wrap`: None
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
- `deepspeed`: None
- `label_smoothing_factor`: 0.0
- `optim`: adamw_torch
- `optim_args`: None
- `adafactor`: False
- `group_by_length`: False
- `length_column_name`: length
- `ddp_find_unused_parameters`: None
- `ddp_bucket_cap_mb`: None
- `ddp_broadcast_buffers`: False
- `dataloader_pin_memory`: True
- `dataloader_persistent_workers`: False
- `skip_memory_metrics`: True
- `use_legacy_prediction_loop`: False
- `push_to_hub`: False
- `resume_from_checkpoint`: True
- `hub_model_id`: None
- `hub_strategy`: every_save
- `hub_private_repo`: False
- `hub_always_push`: False
- `gradient_checkpointing`: False
- `gradient_checkpointing_kwargs`: None
- `include_inputs_for_metrics`: False
- `eval_do_concat_batches`: True
- `fp16_backend`: auto
- `push_to_hub_model_id`: None
- `push_to_hub_organization`: None
- `mp_parameters`: 
- `auto_find_batch_size`: False
- `full_determinism`: False
- `torchdynamo`: None
- `ray_scope`: last
- `ddp_timeout`: 1800
- `torch_compile`: False
- `torch_compile_backend`: None
- `torch_compile_mode`: None
- `dispatch_batches`: None
- `split_batches`: None
- `include_tokens_per_second`: False
- `include_num_input_tokens_seen`: False
- `neftune_noise_alpha`: None
- `optim_target_modules`: None
- `batch_eval_metrics`: False
- `eval_on_start`: False
- `use_liger_kernel`: False
- `eval_use_gather_object`: False
- `batch_sampler`: batch_sampler
- `multi_dataset_batch_sampler`: proportional

</details>

### Training Logs
<details><summary>Click to expand</summary>

| Epoch   | Step | Training Loss | loss   | beatai-dev_max_accuracy |
|:-------:|:----:|:-------------:|:------:|:-----------------------:|
| 0       | 0    | -             | -      | 0.8308                  |
| 0.1471  | 10   | 1.056         | -      | -                       |
| 0.2941  | 20   | 1.0992        | -      | -                       |
| 0.4412  | 30   | 1.1678        | -      | -                       |
| 0.5882  | 40   | 1.1586        | -      | -                       |
| 0.7353  | 50   | 1.1777        | 2.0793 | 0.8291                  |
| 0.8824  | 60   | 1.1344        | -      | -                       |
| 1.0294  | 70   | 1.0578        | -      | -                       |
| 1.1765  | 80   | 1.0981        | -      | -                       |
| 1.3235  | 90   | 1.1216        | -      | -                       |
| 1.4706  | 100  | 1.0436        | 2.0826 | 0.8283                  |
| 1.6176  | 110  | 1.0422        | -      | -                       |
| 1.7647  | 120  | 1.0857        | -      | -                       |
| 1.9118  | 130  | 1.0502        | -      | -                       |
| 2.0588  | 140  | 1.0363        | -      | -                       |
| 2.2059  | 150  | 1.081         | 2.0763 | 0.8316                  |
| 2.3529  | 160  | 1.1764        | -      | -                       |
| 2.5     | 170  | 1.0393        | -      | -                       |
| 2.6471  | 180  | 0.9586        | -      | -                       |
| 2.7941  | 190  | 1.0537        | -      | -                       |
| 2.9412  | 200  | 1.0313        | 2.0645 | 0.8325                  |
| 3.0882  | 210  | 1.0401        | -      | -                       |
| 3.2353  | 220  | 1.0389        | -      | -                       |
| 3.3824  | 230  | 1.0225        | -      | -                       |
| 3.5294  | 240  | 1.0131        | -      | -                       |
| 3.6765  | 250  | 0.9565        | 2.0705 | 0.8308                  |
| 3.8235  | 260  | 1.0059        | -      | -                       |
| 3.9706  | 270  | 0.9629        | -      | -                       |
| 4.1176  | 280  | 0.9546        | -      | -                       |
| 4.2647  | 290  | 0.989         | -      | -                       |
| 4.4118  | 300  | 1.0573        | 2.0514 | 0.8375                  |
| 4.5588  | 310  | 0.894         | -      | -                       |
| 4.7059  | 320  | 1.0082        | -      | -                       |
| 4.8529  | 330  | 0.969         | -      | -                       |
| 5.0     | 340  | 0.9187        | -      | -                       |
| 5.1471  | 350  | 0.9034        | 2.0663 | 0.8350                  |
| 5.2941  | 360  | 0.9043        | -      | -                       |
| 5.4412  | 370  | 0.9517        | -      | -                       |
| 5.5882  | 380  | 1.0272        | -      | -                       |
| 5.7353  | 390  | 0.95          | -      | -                       |
| 5.8824  | 400  | 0.8288        | 2.0400 | 0.8367                  |
| 6.0294  | 410  | 0.9809        | -      | -                       |
| 6.1765  | 420  | 0.8776        | -      | -                       |
| 6.3235  | 430  | 0.9744        | -      | -                       |
| 6.4706  | 440  | 0.9982        | -      | -                       |
| 6.6176  | 450  | 0.9076        | 2.0429 | 0.8350                  |
| 6.7647  | 460  | 0.8792        | -      | -                       |
| 6.9118  | 470  | 0.787         | -      | -                       |
| 7.0588  | 480  | 0.9506        | -      | -                       |
| 7.2059  | 490  | 0.927         | -      | -                       |
| 7.3529  | 500  | 0.9464        | 2.0487 | 0.8316                  |
| 7.5     | 510  | 0.886         | -      | -                       |
| 7.6471  | 520  | 0.9142        | -      | -                       |
| 7.7941  | 530  | 0.8741        | -      | -                       |
| 7.9412  | 540  | 0.8703        | -      | -                       |
| 8.0882  | 550  | 0.8947        | 2.0411 | 0.8333                  |
| 8.2353  | 560  | 0.8742        | -      | -                       |
| 8.3824  | 570  | 0.8083        | -      | -                       |
| 8.5294  | 580  | 0.9134        | -      | -                       |
| 8.6765  | 590  | 0.8197        | -      | -                       |
| 8.8235  | 600  | 0.8253        | 2.0272 | 0.8367                  |
| 8.9706  | 610  | 0.8665        | -      | -                       |
| 9.1176  | 620  | 0.8853        | -      | -                       |
| 9.2647  | 630  | 0.7566        | -      | -                       |
| 9.4118  | 640  | 0.9101        | -      | -                       |
| 9.5588  | 650  | 0.801         | 2.0243 | 0.8350                  |
| 9.7059  | 660  | 0.8551        | -      | -                       |
| 9.8529  | 670  | 0.8748        | -      | -                       |
| 10.0    | 680  | 0.9798        | -      | -                       |
| 10.1471 | 690  | 1.0544        | -      | -                       |
| 10.2941 | 700  | 1.2077        | 2.0128 | 0.8367                  |
| 10.4412 | 710  | 1.0386        | -      | -                       |
| 10.5882 | 720  | 1.0508        | -      | -                       |
| 10.7353 | 730  | 1.0063        | -      | -                       |
| 10.8824 | 740  | 1.0758        | -      | -                       |
| 11.0294 | 750  | 1.1552        | 2.0031 | 0.8367                  |
| 11.1765 | 760  | 1.0259        | -      | -                       |
| 11.3235 | 770  | 1.0724        | -      | -                       |
| 11.4706 | 780  | 1.0524        | -      | -                       |
| 11.6176 | 790  | 0.9957        | -      | -                       |
| 11.7647 | 800  | 1.0697        | 2.0022 | 0.8367                  |
| 11.9118 | 810  | 1.0544        | -      | -                       |
| 12.0588 | 820  | 1.0762        | -      | -                       |
| 12.2059 | 830  | 1.0858        | -      | -                       |
| 12.3529 | 840  | 1.0418        | -      | -                       |
| 12.5    | 850  | 1.0041        | 1.9936 | 0.8392                  |
| 12.6471 | 860  | 0.998         | -      | -                       |
| 12.7941 | 870  | 1.0737        | -      | -                       |
| 12.9412 | 880  | 1.0637        | -      | -                       |
| 13.0882 | 890  | 0.9689        | -      | -                       |
| 13.2353 | 900  | 1.001         | 1.9818 | 0.8392                  |
| 13.3824 | 910  | 1.0418        | -      | -                       |
| 13.5294 | 920  | 1.0097        | -      | -                       |
| 13.6765 | 930  | 1.0244        | -      | -                       |
| 13.8235 | 940  | 1.0383        | -      | -                       |
| 13.9706 | 950  | 1.034         | 1.9798 | 0.8367                  |
| 14.1176 | 960  | 0.9609        | -      | -                       |
| 14.2647 | 970  | 1.049         | -      | -                       |
| 14.4118 | 980  | 1.0012        | -      | -                       |
| 14.5588 | 990  | 0.9008        | -      | -                       |
| 14.7059 | 1000 | 1.0131        | 1.9741 | 0.8384                  |
| 14.8529 | 1010 | 0.9714        | -      | -                       |
| 15.0    | 1020 | 0.9987        | -      | -                       |
| 15.1471 | 1030 | 1.1139        | -      | -                       |
| 15.2941 | 1040 | 1.005         | -      | -                       |
| 15.4412 | 1050 | 0.9074        | 1.9761 | 0.8359                  |
| 15.5882 | 1060 | 0.9298        | -      | -                       |
| 15.7353 | 1070 | 0.9335        | -      | -                       |
| 15.8824 | 1080 | 0.9445        | -      | -                       |
| 16.0294 | 1090 | 1.0087        | -      | -                       |
| 16.1765 | 1100 | 0.9187        | 1.9679 | 0.8384                  |
| 16.3235 | 1110 | 0.8502        | -      | -                       |
| 16.4706 | 1120 | 0.9924        | -      | -                       |
| 16.6176 | 1130 | 0.9982        | -      | -                       |
| 16.7647 | 1140 | 0.9643        | -      | -                       |
| 16.9118 | 1150 | 0.9491        | 1.9727 | 0.8333                  |
| 17.0588 | 1160 | 0.9801        | -      | -                       |
| 17.2059 | 1170 | 0.9374        | -      | -                       |
| 17.3529 | 1180 | 0.8309        | -      | -                       |
| 17.5    | 1190 | 0.9524        | -      | -                       |
| 17.6471 | 1200 | 0.886         | 1.9797 | 0.8350                  |
| 17.7941 | 1210 | 0.9026        | -      | -                       |
| 17.9412 | 1220 | 0.8859        | -      | -                       |
| 18.0882 | 1230 | 0.8745        | -      | -                       |
| 18.2353 | 1240 | 0.9474        | -      | -                       |
| 18.3824 | 1250 | 0.878         | 1.9737 | 0.8342                  |
| 18.5294 | 1260 | 0.8372        | -      | -                       |
| 18.6765 | 1270 | 0.833         | -      | -                       |
| 18.8235 | 1280 | 0.9648        | -      | -                       |
| 18.9706 | 1290 | 0.918         | -      | -                       |
| 19.1176 | 1300 | 0.9588        | 1.9669 | 0.8359                  |
| 19.2647 | 1310 | 1.0334        | -      | -                       |
| 19.4118 | 1320 | 0.8347        | -      | -                       |
| 19.5588 | 1330 | 0.828         | -      | -                       |
| 19.7059 | 1340 | 0.9117        | -      | -                       |
| 19.8529 | 1350 | 0.9123        | 1.9666 | 0.8350                  |
| 20.0    | 1360 | 0.8538        | -      | -                       |

</details>

### Framework Versions
- Python: 3.8.18
- Sentence Transformers: 3.1.1
- Transformers: 4.45.0
- PyTorch: 1.13.1+cu117
- Accelerate: 0.34.2
- Datasets: 3.0.1
- Tokenizers: 0.20.0

## Citation

### BibTeX

#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}
```

#### TripletLoss
```bibtex
@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->