setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'The answer provided directly relates to the question asked and is well-supported
    by the document, which explains the percentage in the response status column as
    the total amount of successful completion of response actions. The answer is concise
    and specific to the query.


    Final evaluation:'
- text: 'Evaluation:

    The answer states that the provided information does not cover the specific query,
    suggesting referring to additional sources or providing more context. However,
    the document does cover the process of enabling and configuring Endpoint controls
    and mentions specific features under Endpoint controls like Device Control, Personal
    Firewall Control, and Full Disk Encryption Visibility. The document does not explicitly
    state the "purpose" of Endpoint controls, but it is evident from the listed features
    that these controls are for managing device control, firewall settings, and disk
    encryption visibility. Therefore, the answer is not well-supported by the document
    and fails to address the specific question adequately.


    Final evaluation:'
- text: 'Reasoning:

    1. **Context Grounding**: The answer is supported by the provided document where
    it is mentioned that the On-Site Collector Agent collects logs and forwards them
    to <ORGANIZATION> XDR.

    2. **Relevance**: The purpose of the <ORGANIZATION> XDR On-Site Collector Agent
    is indeed to collect and securely forward logs.

    3. **Conciseness**: The answer is concise and directly addresses the specific
    question asked without unnecessary information.

    4. **Specificity**: The answer is specific to the question regarding the purpose
    of the On-Site Collector Agent, without being too general.

    5. **Key/Value/Event Name**: Although the answer does not include keys or values
    from the document, it is not necessary for this specific question about the purpose
    of the agent.


    The answer meets allthe criteria effectively.


    Final evaluation:'
- text: "The provided answer does not align well with the document. Here's a detailed\
    \ analysis of the evaluation criteria:\n\n1. **Context Grounding**: The answer\
    \ does not seem to be backed up by the specifics provided in the document. The\
    \ document describes settings around making sensors stale, archived, or deleted\
    \ and associated email notifications, but it does not explicitly mention a checkbox\
    \ for email notifications in the Users section.\n\n2. **Relevance**: The answer\
    \ does not correctly address the specific query about the checkbox in the Users\
    \ section as per the document content. \n\n3. **Conciseness**: While the answer\
    \ is concise, it is not directly supported by the content of the document, making\
    \ it irrelevant.\n\n4. **Specificity**: The answer lacks specific details or a\
    \ direct quote from the document that mentions the Users section checkbox.\n\n\
    5. **Accuracy in Key/Value/Event Name**: The document does not provide details\
    \ about a checkbox for email notifications in the Users section, thus the key/value/event\
    \ name aspect is also not correctly covered.\n\nBased on these points, the answer\
    \ provided fails to meet the necessary criteria.\n\nFinal evaluation: ****"
- text: "**Reasoning**:\n\n1. **Context Grounding**: The answer does not match the\
    \ context provided in the document. The document specifies different URLs for\
    \ images related to DNS queries and connection queries.\n   \n2. **Relevance**:\
    \ The answer is not relevant to the specific question asked. The question asks\
    \ for the URL of the image for the second query, which is clearly provided in\
    \ the document but not correctly retrieved in the answer.\n\n3. **Conciseness**:\
    \ The answer is concise but incorrect, making it not useful.\n\n4. **Specificity**:\
    \ The answer lacks accuracy, which is critical for answering the specific question.\
    \ It provides an incorrect URL.\n\n5. **Key, Value, Event Name**: Since the question\
    \ is about a specific URL, correctness of the key/value is crucial, which the\
    \ answer fails to provide.\n\n**Final evaluation**:"
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.6956521739130435
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|:------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>'Evaluation:\nThe answer provides details about Ennita Manyumwa and effectively connects her personal actions to the broader context of the fight against AIDS, specifically highlighting her significance as a symbol of prevention strategies emphasized by the Bush administration. All these points are directly supported by the provided document.\n\nThe final evaluation:'</li><li>'**Reasoning:**\n1. **Context Grounding:** The answer is thoroughly supported by the provided document, listing out the benefits exactly as described by the author.\n2. **Relevance:** The answer remains focused on the benefits experienced by the author from their regular yoga practice, aligning closely with the question.\n3. **Conciseness:** The answer is comprehensive and succinct, encapsulating all salient points without straying into unrelated content.\n\n**Final Evaluation: **'</li><li>'Reasoning:\n1. Context Grounding: The answer is grounded in the provided document. It references the progression of research in germ-free-life at Notre Dame University, leading to the establishment of the Lobund Institute.\n2. Relevance: The answer directly addresses the question of which institute was created as a result of the germ-free-life research at Notre Dame.\n3. Conciseness: The answer is concise and clearly states the outcome of the research.\n\nFinal Evaluation:'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
| 0     | <ul><li>'### Evaluation\n\n1. **Context Grounding:** The answer is closely aligned with the provided document, delivering a detailed and step-by-step procedure for enabling approval for appointment bookings.\n\n2. **Relevance:** The answer veers slightly off-point; the question asks about clients being unable to book appointments online, whereas the answer primarily focuses on how to set up and manage appointment booking approvals. While the setup of booking approvals could potentially address issues with online appointments, the explanation should explicitly connect to troubleshooting clients\' inability to book.\n\n3. **Conciseness:** The answer is lengthy but thorough. Though comprehensive, it contains extra steps and information beyond solving the specific problem posed by the question.\n\n4. **Correct and Detailed Instructions:** The instructions provided are very detailed and correct according to the document, yet they lack direct relevance to troubleshooting "unable to book" issues.\n\n### Reasoning:\n\nThe answer provides detailed instructions grounded in the document but somewhat sidesteps the main question of troubleshooting clients\' inability to book appointments. While the provided steps do explain enabling manual booking approvals, they do not directly address potential issues causing clients to be unable to book appointments online.\n\n### Final Result:\n\n****'</li><li>'Evaluation:\n1. Context Grounding: The answer is well-supported by the provided document, effectively utilizing the information about using strong words, cutting unnecessary words, avoiding clichés, and making use of descriptive language.\n2. Relevance: The answer directly addresses the question "How to Improve Your Writing Skills" by recommending practical advice that is consistent with the guidance provided in the document.\n3. Conciseness: The answer is clear and to the point, offering succinct advice without deviating into unnecessary information.\n\nOverall, the answer meets all the evaluation criteria effectively.\n\nFinal evaluation:'</li><li>'**Reasoning:**\n\n1. **Context Grounding**: The answer mentions benefits like "unapologetic \'me\' time," "improved health," "self-growth," "increased patience," and "acceptance of daily changes," which are indeed supported by and directly grounded in the provided document. However, the claim that the author experienced "the ability to levitate during meditation sessions" is not found or supported by the document. This is a clear instance of hallucination.\n\n2. **Relevance**: The answer is generally relevant to the question, as it lists several benefits the author experienced from regular yoga practice. However, the inclusion of levitation, which is not mentioned in the document, detracts from the relevance.\n\n3. **Conciseness**: The answer is relatively concise but the inclusion of false information (levitation) detracts fromits clarity and precision.\n\n**Final Result**:'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.6957   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_cybereason_gpt-4o_cot-few_shot-instructions_remove_final_evaluation_e1_on")
# Run inference
preds = model("The answer provided directly relates to the question asked and is well-supported by the document, which explains the percentage in the response status column as the total amount of successful completion of response actions. The answer is concise and specific to the query.

Final evaluation:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 11  | 87.4724 | 221 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 311                   |
| 1     | 322                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0006 | 1    | 0.2074        | -               |
| 0.0316 | 50   | 0.2614        | -               |
| 0.0632 | 100  | 0.2527        | -               |
| 0.0948 | 150  | 0.2362        | -               |
| 0.1263 | 200  | 0.1995        | -               |
| 0.1579 | 250  | 0.1809        | -               |
| 0.1895 | 300  | 0.1319        | -               |
| 0.2211 | 350  | 0.126         | -               |
| 0.2527 | 400  | 0.0964        | -               |
| 0.2843 | 450  | 0.0609        | -               |
| 0.3159 | 500  | 0.027         | -               |
| 0.3474 | 550  | 0.0246        | -               |
| 0.3790 | 600  | 0.0185        | -               |
| 0.4106 | 650  | 0.0111        | -               |
| 0.4422 | 700  | 0.0114        | -               |
| 0.4738 | 750  | 0.0182        | -               |
| 0.5054 | 800  | 0.0135        | -               |
| 0.5370 | 850  | 0.0114        | -               |
| 0.5685 | 900  | 0.0067        | -               |
| 0.6001 | 950  | 0.0048        | -               |
| 0.6317 | 1000 | 0.0072        | -               |
| 0.6633 | 1050 | 0.0042        | -               |
| 0.6949 | 1100 | 0.0036        | -               |
| 0.7265 | 1150 | 0.0025        | -               |
| 0.7581 | 1200 | 0.0055        | -               |
| 0.7896 | 1250 | 0.0056        | -               |
| 0.8212 | 1300 | 0.0049        | -               |
| 0.8528 | 1350 | 0.0034        | -               |
| 0.8844 | 1400 | 0.0036        | -               |
| 0.9160 | 1450 | 0.0029        | -               |
| 0.9476 | 1500 | 0.0036        | -               |
| 0.9792 | 1550 | 0.0034        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->