setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'The percentage in the response status column indicates the total amount of
    successful completion of response actions.


    Reasoning:

    1. **Context Grounding**: The answer is well-supported by the document which states,
    "percentage indicates the total amount of successful completion of response actions."

    2. **Relevance**: The answer directly addresses the specific question asked about
    what the percentage in the response status column indicates.

    3. **Conciseness**: The answer is succinct and to the point without unnecessary
    information.

    4. **Specificity**: The answer is specific to what is being asked, detailing exactly
    what the percentage represents.

    5. **Accuracy**: The answer provides the correct key/value as per the document.


    Final result:'
- text: 'Reasoning:

    1. **Context Grounding**: The provided document does outline steps to enable Endpoint
    controls but doesn''t explicitly state their purpose.

    2. **Relevance**: The answer acknowledges the lack of specific information in
    the document about the purpose of Endpoint controls.

    3. **Conciseness**: The answer is concise, directly addressing the lack of information.

    4. **Specificity**: The answer directly states that the document doesn''t answer
    the query, suggesting further sources should be checked.

    5. **Detailed Key/Value/Event Name Check**: These elements do not apply to this
    specific question.


    Considering the criteria, the answer is accurate in indicating the document does
    not provide the purpose of Endpoint controls and suggests looking for additional
    sources.


    Final Result:'
- text: 'Reasoning:


    1. **Context Grounding**: The answer refers to using the <ORGANIZATION> XDR to
    collect and forward logs, but it does not directly mention the <ORGANIZATION>
    XDR On-Site Collector Agent, although it is tangentially related.

    2. **Relevance**: The question specifically inquires about the purpose of the
    <ORGANIZATION> XDR On-Site Collector Agent, not the general functionality of <ORGANIZATION>
    XDR. The answer provided does not address the agent itself.

    3. **Conciseness**: The answer provided is concise but unfortunately lacks relevance
    to the specific question being asked.

    4. **Specificity**: The answer is too general and doesn''t provide the specific
    purpose of the On-Site Collector Agent.

    5. **Key/Value/Event Name**: The answer does not include any specific key, value,
    or event name that would relate to discussing an On-Site Collector Agent.


    Final result: ****'
- text: "Reasoning:\n\n1. **Context Grounding**: The provided answer mentions the\
    \ purpose of the <ORGANIZATION_2> email notifications checkbox in relation to\
    \ enabling or disabling email notifications for users. However, the document explicitly\
    \ states that notifications about stale and archived sensors are managed separately\
    \ from other email preferences. The checkbox in the Users section determines whether\
    \ users receive these specific notifications, which indicates a more precise purpose.\n\
    \  \n2. **Relevance**: The response does relate to the question but lacks specificity\
    \ about the type of notifications (stale/archived sensors) governed by the checkbox.\
    \ It also fails to mention that these notifications are managed independently\
    \ of other email preferences.\n  \n3. **Conciseness**: The answer is concise but\
    \ could be clearer about the specific type of notifications and their management.\n\
    \  \n4. **Specificity**: The answer is somewhat general and does not fully capture\
    \ the detailed function of the checkbox as described in the document.\n  \n5.\
    \ **Correct Key/Value/Event Name**: The answer correctly identifies the purpose\
    \ of the checkbox but does not reflect the detailed context provided in the document\
    \ regarding specific notifications (stale/archived sensors).\n\nFinal Result:"
- text: "The provided answer \"..\\/..\\/_images\\/hunting_http://www.flores.net/\"\
    \ does not match the correct URL as per the document content for the second query.\n\
    \n**Reasoning:**\n1. **Context Grounding:**\n   - The URL provided \"..\\/..\\\
    /_images\\/hunting_http://www.flores.net/\" is not found in the provided document.\n\
    \   - Instead, the correct URL as per the document for Query 2 is \"..\\/..\\\
    /_images\\/hunting_http://miller.co\".\n\n2. **Relevance:**\n   - The answer provided\
    \ does not correspond to the specific question asked, which was about the URL\
    \ for the second query. It deviates from the document and is incorrect.\n\n3.\
    \ **Conciseness:**\n   - The answer does not provide any extraneous information,\
    \ but being incorrect, it fails at providing the relevant and necessary detail\
    \ concisely.\n\n4. **Specificity:**\n   - The answer is specific but incorrect.\
    \ It provides a URL, but not the right one as required.\n\n5. **Accuracy of key/value/event\
    \ name:**\n   - The correct event (image URL) for the second query is \"..\\/..\\\
    /_images\\/hunting_http://miller.co\"according to the document.\n\nFinal result:\
    \ ****"
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.6901408450704225
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0     | <ul><li>'The answer provided does not address the question asked about the lifespan of John Augustine Zahm. It discusses the ownership of the land where the Bell and Gemmell tannery was located, which is unrelated to the specific question.\n\nReasoning:\n1. **Context Grounding**: The information about John Buchanan and the Bell and Gemmell tannery is grounded in the provided document but does not relate to John Augustine Zahm or his lifespan.\n2. **Relevance**: The answer does not address the question regarding John Augustine Zahm’s lifespan at all. It is completely off-topic.\n3. **Conciseness**: While the provided answer is concise, it is entirely irrelevant to the question concerning John Augustine Zahm’s lifespan. \n\nFinal Result:'</li><li>'Reasoning: \n\n1. **Context Grounding**: The answer correctly identifies the federal funds rate as the main tool of conventional monetary policy in the USA. This statement is well-supported by the provided document, which specifies that the federal funds rate is the primary mechanism used in conventional monetary policy in the USA, and further explains its role in interbank lending.\n \n2. **Relevance**: The question asks specifically about the main tool of conventional monetary policy in the USA. The answer correctly states that the federal funds rate is this tool. However, there is an error in attributing the decision of the federal funds rate to Congress when in fact it is determined by the Federal Reserve.\n\n3. **Conciseness**: The answer is brief and directly addresses the question without unnecessary information. However, the incorrect statement about Congress deciding the rate detracts from its overall clarity and correctness.\n\nFinal Result:'</li><li>'Reasoning:\n1. **Context Grounding**: The answer partially reflects the document\'s suggestions about using opponents\' momentum and techniques to trip them. However, it introduces methods like "flailing wildly" and "jumping above punches," which are not grounded in the document provided.\n2. **Relevance**: The answer does not fully align with the document\'s practical and structured advice for takedowns. It diverges into suggestions that are unrealistic and not mentioned in the document.\n3. **Conciseness**: The answer is overly detailed in unnecessary ways, such as the emphasis on slow execution and crossing legs, which are not key points mentioned in the document provided. Rather, the document stresses quick and committed actions.\n\nFinal Result:'</li></ul> |
| 1     | <ul><li>"Reasoning:\n1. **Context Grounding:** The answer is directly supported by the content in the document. It correctly identifies the disagreement as being about the duration of the payroll tax cut and highlights the specific durations preferred by both parties.\n2. **Relevance:** The answer precisely addresses the question by focusing solely on the disagreement about the payroll tax cut's duration between Democrats and Republicans.\n3. **Conciseness:** The answer is succinct and to the point. It avoids extraneous information, sticking strictly to explaining the core disagreement.\n\nResult:"</li><li>'**Reasoning:**\n\n1. **Context Grounding:** The provided answer draws directly from the supplied document, mentioning the variety and customizability of templates offered by the organization as well as their qualities, such as including sample content and different features.\n\n2. **Relevance:** The response is related to the inquiry about available blog post templates. However, while it accurately includes information about the variety of templates and customization options, it should focus more narrowly on templates specifically for blog posts.\n\n3. **Conciseness:** The answer is somewhat verbose. It includes information that isn’t strictly necessary to answer the question, such as specifics about the overall number of templates and customization options, instead of focusing solely on blog templates.\n\n4. **Correct Instructions:** The answer does correctly instruct the reader on how to choose a template and its customization, but it lacks specific emphasis on the blog aspect of the templates which is critical to the question.\n\n**Evaluation Result:**'</li><li>'Reasoning:\n\n1. Context Grounding: The document clearly identifies "Father Joseph Carrier" as the person holding the professorship of Chemistry and Physics at Notre Dame, not "Father Josh Carrier."\n2. Relevance: The answer provided states that "Father Josh Carrier" held the professorship, which is incorrect based on the provided document.\n3. Conciseness: While the answer is concise, it is not factually accurate.\n\nFinal Result:'</li></ul>                                                                                                                                                                                                                                                                                                                                                                         |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.6901   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_cybereason_gpt-4o_cot-instructions_remove_final_evaluation_e1_one_out_172")
# Run inference
preds = model("The percentage in the response status column indicates the total amount of successful completion of response actions.

Reasoning:
1. **Context Grounding**: The answer is well-supported by the document which states, \"percentage indicates the total amount of successful completion of response actions.\"
2. **Relevance**: The answer directly addresses the specific question asked about what the percentage in the response status column indicates.
3. **Conciseness**: The answer is succinct and to the point without unnecessary information.
4. **Specificity**: The answer is specific to what is being asked, detailing exactly what the percentage represents.
5. **Accuracy**: The answer provides the correct key/value as per the document.

Final result:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median   | Max |
|:-------------|:----|:---------|:----|
| Word count   | 32  | 103.2508 | 245 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 312                   |
| 1     | 322                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0006 | 1    | 0.2802        | -               |
| 0.0315 | 50   | 0.2628        | -               |
| 0.0631 | 100  | 0.2537        | -               |
| 0.0946 | 150  | 0.2553        | -               |
| 0.1262 | 200  | 0.2546        | -               |
| 0.1577 | 250  | 0.2477        | -               |
| 0.1893 | 300  | 0.2189        | -               |
| 0.2208 | 350  | 0.1745        | -               |
| 0.2524 | 400  | 0.1426        | -               |
| 0.2839 | 450  | 0.0785        | -               |
| 0.3155 | 500  | 0.0569        | -               |
| 0.3470 | 550  | 0.0402        | -               |
| 0.3785 | 600  | 0.0325        | -               |
| 0.4101 | 650  | 0.023         | -               |
| 0.4416 | 700  | 0.0098        | -               |
| 0.4732 | 750  | 0.017         | -               |
| 0.5047 | 800  | 0.0129        | -               |
| 0.5363 | 850  | 0.0087        | -               |
| 0.5678 | 900  | 0.0027        | -               |
| 0.5994 | 950  | 0.0058        | -               |
| 0.6309 | 1000 | 0.0044        | -               |
| 0.6625 | 1050 | 0.0046        | -               |
| 0.6940 | 1100 | 0.0052        | -               |
| 0.7256 | 1150 | 0.0021        | -               |
| 0.7571 | 1200 | 0.0015        | -               |
| 0.7886 | 1250 | 0.0013        | -               |
| 0.8202 | 1300 | 0.0014        | -               |
| 0.8517 | 1350 | 0.0012        | -               |
| 0.8833 | 1400 | 0.0013        | -               |
| 0.9148 | 1450 | 0.0012        | -               |
| 0.9464 | 1500 | 0.0013        | -               |
| 0.9779 | 1550 | 0.0013        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->