setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning:

    The provided answer detailed the performance review process in a coherent manner.
    However, upon comparison with the document, several issues arise:


    1. **Context Grounding**: The answer maintains a general alignment with the document
    but shows misalignment in its content by including phrases like "TORGANIZATION_11ey
    can also ORGANIZATION_11Richard Carlson coffee witORGANIZATION_11 tORGANIZATION_11eir
    teammates" which makes no logical sense and disrupts the flow of information.


    2. **Relevance**: The answer mostly stays relevant to the question but includes
    an unnecessary reference to an incomprehensible phrase about coffee with teammates.


    3. **Conciseness**: The answer is relatively clear but could have been more concise
    by avoiding unnecessary and confusing phrases.


    4. **Specificity**: The answer should have focused more on the specific steps
    mentioned in the document, such as providing links to forms and detailing the
    exact process flow.


    Because of these issues, the intricacies of the proper feedback process were blurred
    by inadequate information processing, leading to a suboptimal response.


    Evaluation:'
- text: 'Reasoning:

    1. **Context Grounding**: The answer appropriately uses information from the document
    to address the query. The email provided (finance@ORGANIZATION_2.<89312988>) matches
    the reference in Document 1.


    2. **Relevance**: The answer is relevant to the question and does not deviate
    into unrelated topics. It directly addresses the contact point for questions on
    travel reimbursement.


    3. **Conciseness**: The answer is clear and to the point, effectively communicating
    the necessary information without unnecessary details.


    4. **Specificity**: The email address given in the answer is taken directly from
    the provided document, ensuring the response is specific and accurate.


    5. **Overall Completeness**: The answer meets all the criteria mentioned and does
    not attempt to address unrelated issues or give overly general information.


    Final evaluation:'
- text: "Reasoning:\n1. **Context Grounding**: The provided answer includes elements\
    \ that are derived from the documents, such as the importance of thinking about\
    \ someone's tenure at the company and flagging issues to Kristin Figueroa. However,\
    \ it misses some nuances and specifics that are present in the documents, such\
    \ as the detailed steps involving feedback, analyzing problems, and specific actions\
    \ to take if a situation does not improve.\n\n2. **Relevance**: The answer is\
    \ mostly relevant to the question but introduces concepts like work-life balance,\
    \ which, while important, are not directly emphasized in the provided documents\
    \ for addressing the specific improvement of the situation between team leads\
    \ and employees.\n\n3. **Conciseness**: The answer is somewhat concise yet fails\
    \ to be thorough in detailing all actionable steps outlined in the documents.\n\
    \n4. **Completeness**: The documents provide a more structured approach including\
    \ specific steps like analyzing the problem, trying to fix it together, and making\
    \ timely decisions if necessary, which are not fully captured in the answer.\n\
    \n5. **Specificity**: The answer lacks detailed procedural aspects such as asking\
    \ for help at the analysis stage and specific pointers for underperformance or\
    \ ceased growth, which are discussed in the provided documents.\n\nOverall, the\
    \ answer could have been more detailed and varied in its approach to fully encapsulate\
    \ the directives provided by the documents. \n\nEvaluation:"
- text: 'Reasoning:

    The provided answer addresses the question directly, making it clear that flirting
    with female colleagues is considered harassment and is not allowed. Here’s a breakdown
    based on the criteria:


    1. **Context Grounding**: The answer is well-supported by the documents, which
    explicitly state that any form of sexual harassment, including behaviors such
    as flirting, making others feel uncomfortable, or unsafe, is prohibited.


    2. **Relevance**: The answer stays relevant to the question asked, focusing entirely
    on the topic of flirting in the workplace and explaining its implications under
    the company policy.


    3. **Conciseness**: The answer is concise, directly stating the consequence of
    flirting without veering off into unrelated subjects. It aligns with the policies
    mentioned in the provided documents.


    4. **Specificity**: The answer includes specific details from the documents, mentioning
    that such behaviors are taken seriously and will result in appropriate disciplinary
    actions, thus ensuring clarity.


    5. **Completeness**: The answer is complete; it covers not only the prohibition
    but also the reason behind it—that it creates discomfort or an unsafe feeling
    among colleagues.


    Therefore, considering the provided criteria and evaluation points:


    Final result:'
- text: "Reasoning:\n1. **Context Grounding**: The answer is not well-grounded in\
    \ the provided documents. It generally mentions why personal relationships are\
    \ important at work but does not specifically refer to the document which mentions\
    \ investing in personal relationships helps in \"so many ways\" (Document 1).\n\
    \n2. **Relevance**: While the answer is related to the question, it includes additional\
    \ information that goes beyond what is specifically mentioned in the documents,\
    \ such as emotional support and stress reduction.\n\n3. **Conciseness**: The answer\
    \ is somewhat verbose and includes information that is not strictly necessary\
    \ to answer the question as per the documents provided. \n\n4. **Specificity**:\
    \ The answer is fairly general and does not detail the specific reasons mentioned\
    \ in the documents. \n\nGiven these issues, the answer does not sufficiently meet\
    \ the criteria for a good answer.\n\nFinal annotation: \nEvaluation:"
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.7313432835820896
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|:------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>'Reasoning: \nThe answer provided is directly relevant to the question about how to cancel a trade in Yahoo Fantasy Football after it has been accepted. It states the specific requirement that 4 out of 10 team owners need to vote to veto the trade within 48 hours for it to be cancelled. This information is well-supported by the document provided. The answer is concise and to the point, addressing the specific question without adding unnecessary details.\n\nFinal result:'</li><li>'Reasoning:\nThe answer correctly identifies Joan Gaspart as the individual who resigned from the presidency of Barcelona after the team\'s poor showing in the 2003 season. This is directly supported by the document, which explicitly states that "club president Joan Gaspart resigned, his position having been made completely untenable by such a disastrous season on top of the club\'s overall decline in fortunes since he became president three years prior." The answer is concise and directly relevant to the question without including any extraneous information.\n\nEvaluation:'</li><li>"Reasoning:\n1. **Context Grounding**: The answer addresses the support for employee wellness and highlights the Headspace app, which matches the content in the documents that describe ORGANIZATION reimbursing Headspace subscriptions for employees.\n2. **Relevance**: The answer directly addresses the question by discussing the support systems in place for employee wellness, particularly the Headspace subscription.\n3. **Conciseness**: The answer is concise and directly provides the relevant information without unnecessary details.\n4. **Direct Response**: The answer does not attempt to extrapolate beyond what is provided in the document and sticks to the information available.\n5. **Specificity**: The answer covers the specifics about the Headspace subscription being reimbursed monthly, aligning well with the document's details.\n6. **Additional Information**: While the answer is correct, including a mention of the fitness subscription as another wellness support option would have been beneficial if provided in the document.\n\nOverall, the answer is well-supported, relevant, and concise, thus appropriately addressing the question based on the provided document.\n\nFinal annotation: \nEvaluation:"</li></ul> |
| 0     | <ul><li>'Reasoning:\nThe answer provided is mostly well-supported by the document and directly addresses the question of how to care for snails. It includes several key points such as providing a well-ventilated tank, using moist peat or soil as substrate, adding climbing objects, and providing calcium sources like cuttlefish bone or eggshells. It also mentions feeding snails fresh fruits and vegetables and maintaining a small water dish—all of which align with the document. \n\nHowever, the answer introduces a point that snails enjoy listening to classical music, which is not mentioned or supported by the document. This inclusion is not grounded in the provided information, and it introduces unnecessary and possibly distracting information.\n\nFinal Evaluation: \n\nEvaluation:'</li><li>'Reasoning:\nThe answer "James Marcia divides questions into five categories" is incorrect according to the information provided in the document. The document clearly states that James Marcia\'s questions are divided into three categories: occupation, ideology, and interpersonal relationships. Thus, the response does not accurately reflect the document\'s content and fails to address the specific question asked.\nEvaluation:'</li><li>'Reasoning:\nThe answer provided is "The Kroc Institute at Notre Dame focuses on International Peace Studies." This response relates to information provided in the document, specifically about the Kroc Institute. However, the question asked is about "What company did Ray Kroc own?" The answer does not address the specific question and insteadprovides irrelevant information.\n\nEvaluation:'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.7313   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_remove_")
# Run inference
preds = model("Reasoning:
1. **Context Grounding**: The answer appropriately uses information from the document to address the query. The email provided (finance@ORGANIZATION_2.<89312988>) matches the reference in Document 1.

2. **Relevance**: The answer is relevant to the question and does not deviate into unrelated topics. It directly addresses the contact point for questions on travel reimbursement.

3. **Conciseness**: The answer is clear and to the point, effectively communicating the necessary information without unnecessary details.

4. **Specificity**: The email address given in the answer is taken directly from the provided document, ensuring the response is specific and accurate.

5. **Overall Completeness**: The answer meets all the criteria mentioned and does not attempt to address unrelated issues or give overly general information.

Final evaluation:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 33  | 97.7083 | 289 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 127                   |
| 1     | 137                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (2, 2)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0015 | 1    | 0.2055        | -               |
| 0.0758 | 50   | 0.2629        | -               |
| 0.1515 | 100  | 0.2514        | -               |
| 0.2273 | 150  | 0.2486        | -               |
| 0.3030 | 200  | 0.2145        | -               |
| 0.3788 | 250  | 0.1535        | -               |
| 0.4545 | 300  | 0.0866        | -               |
| 0.5303 | 350  | 0.0335        | -               |
| 0.6061 | 400  | 0.0101        | -               |
| 0.6818 | 450  | 0.0041        | -               |
| 0.7576 | 500  | 0.0027        | -               |
| 0.8333 | 550  | 0.0024        | -               |
| 0.9091 | 600  | 0.0022        | -               |
| 0.9848 | 650  | 0.0024        | -               |
| 1.0606 | 700  | 0.002         | -               |
| 1.1364 | 750  | 0.0018        | -               |
| 1.2121 | 800  | 0.0016        | -               |
| 1.2879 | 850  | 0.0016        | -               |
| 1.3636 | 900  | 0.0015        | -               |
| 1.4394 | 950  | 0.0014        | -               |
| 1.5152 | 1000 | 0.0015        | -               |
| 1.5909 | 1050 | 0.0013        | -               |
| 1.6667 | 1100 | 0.0013        | -               |
| 1.7424 | 1150 | 0.0013        | -               |
| 1.8182 | 1200 | 0.0012        | -               |
| 1.8939 | 1250 | 0.0013        | -               |
| 1.9697 | 1300 | 0.0013        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->