setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning:

    The provided answer detailed the performance review process in a coherent manner.
    However, upon comparison with the document, several issues arise:


    1. **Context Grounding**: The answer maintains a general alignment with the document
    but shows misalignment in its content by including phrases like "TORGANIZATION_11ey
    can also ORGANIZATION_11Richard Carlson coffee witORGANIZATION_11 tORGANIZATION_11eir
    teammates" which makes no logical sense and disrupts the flow of information.


    2. **Relevance**: The answer mostly stays relevant to the question but includes
    an unnecessary reference to an incomprehensible phrase about coffee with teammates.


    3. **Conciseness**: The answer is relatively clear but could have been more concise
    by avoiding unnecessary and confusing phrases.


    4. **Specificity**: The answer should have focused more on the specific steps
    mentioned in the document, such as providing links to forms and detailing the
    exact process flow.


    Because of these issues, the intricacies of the proper feedback process were blurred
    by inadequate information processing, leading to a suboptimal response.


    Evaluation: Bad'
- text: 'Reasoning:

    1. **Context Grounding**: The answer appropriately uses information from the document
    to address the query. The email provided (finance@ORGANIZATION_2.<89312988>) matches
    the reference in Document 1.


    2. **Relevance**: The answer is relevant to the question and does not deviate
    into unrelated topics. It directly addresses the contact point for questions on
    travel reimbursement.


    3. **Conciseness**: The answer is clear and to the point, effectively communicating
    the necessary information without unnecessary details.


    4. **Specificity**: The email address given in the answer is taken directly from
    the provided document, ensuring the response is specific and accurate.


    5. **Overall Completeness**: The answer meets all the criteria mentioned and does
    not attempt to address unrelated issues or give overly general information.


    Final evaluation: Good'
- text: "Reasoning:\n1. **Context Grounding**: The provided answer includes elements\
    \ that are derived from the documents, such as the importance of thinking about\
    \ someone's tenure at the company and flagging issues to Kristin Figueroa. However,\
    \ it misses some nuances and specifics that are present in the documents, such\
    \ as the detailed steps involving feedback, analyzing problems, and specific actions\
    \ to take if a situation does not improve.\n\n2. **Relevance**: The answer is\
    \ mostly relevant to the question but introduces concepts like work-life balance,\
    \ which, while important, are not directly emphasized in the provided documents\
    \ for addressing the specific improvement of the situation between team leads\
    \ and employees.\n\n3. **Conciseness**: The answer is somewhat concise yet fails\
    \ to be thorough in detailing all actionable steps outlined in the documents.\n\
    \n4. **Completeness**: The documents provide a more structured approach including\
    \ specific steps like analyzing the problem, trying to fix it together, and making\
    \ timely decisions if necessary, which are not fully captured in the answer.\n\
    \n5. **Specificity**: The answer lacks detailed procedural aspects such as asking\
    \ for help at the analysis stage and specific pointers for underperformance or\
    \ ceased growth, which are discussed in the provided documents.\n\nOverall, the\
    \ answer could have been more detailed and varied in its approach to fully encapsulate\
    \ the directives provided by the documents. \n\nEvaluation: Bad"
- text: 'Reasoning:

    The provided answer addresses the question directly, making it clear that flirting
    with female colleagues is considered harassment and is not allowed. Here’s a breakdown
    based on the criteria:


    1. **Context Grounding**: The answer is well-supported by the documents, which
    explicitly state that any form of sexual harassment, including behaviors such
    as flirting, making others feel uncomfortable, or unsafe, is prohibited.


    2. **Relevance**: The answer stays relevant to the question asked, focusing entirely
    on the topic of flirting in the workplace and explaining its implications under
    the company policy.


    3. **Conciseness**: The answer is concise, directly stating the consequence of
    flirting without veering off into unrelated subjects. It aligns with the policies
    mentioned in the provided documents.


    4. **Specificity**: The answer includes specific details from the documents, mentioning
    that such behaviors are taken seriously and will result in appropriate disciplinary
    actions, thus ensuring clarity.


    5. **Completeness**: The answer is complete; it covers not only the prohibition
    but also the reason behind it—that it creates discomfort or an unsafe feeling
    among colleagues.


    Therefore, considering the provided criteria and evaluation points:


    Final result: Good'
- text: "Reasoning:\n1. **Context Grounding**: The answer is not well-grounded in\
    \ the provided documents. It generally mentions why personal relationships are\
    \ important at work but does not specifically refer to the document which mentions\
    \ investing in personal relationships helps in \"so many ways\" (Document 1).\n\
    \n2. **Relevance**: While the answer is related to the question, it includes additional\
    \ information that goes beyond what is specifically mentioned in the documents,\
    \ such as emotional support and stress reduction.\n\n3. **Conciseness**: The answer\
    \ is somewhat verbose and includes information that is not strictly necessary\
    \ to answer the question as per the documents provided. \n\n4. **Specificity**:\
    \ The answer is fairly general and does not detail the specific reasons mentioned\
    \ in the documents. \n\nGiven these issues, the answer does not sufficiently meet\
    \ the criteria for a good answer.\n\nFinal annotation: bad\nEvaluation: Bad"
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.7014925373134329
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0     | <ul><li>'Reasoning:\n1. **Context Grounding**: The answer given is generic and lacks specific context from the provided documents. The documents primarily discuss learning and development budgets and prioritize spending based on the role and responsibilities at the organization, but not explicitly about personal money-saving strategies.\n  \n2. **Relevance**: The answer does not directly address the core content of the question based on the available documents. It gives general advice on money savings and investments rather than touching on specific procedures or policies mentioned in the documents.\n   \n3. **Conciseness**: While the answer is concise, it fails to provide detailed or specific guidance grounded in the given documents. It could be said to provide superficial information without drawing from the provided texts.\n   \n4. **Specificity**: The answer is too broad and general. It lacks specifics from the document content (study budget, decision criteria for approving such budgets, etc.), which would make it more relevant and grounded.\n\n5. **Non-Attempt**: The answer makes an erroneous attempt to respond to a question with information that is not backed by the provided documents. It should not have attempted to answer without adequate relevant support from the content.\n\nGiven these failings, the evaluative result is clear.\nEvaluation: Bad'</li><li>"Reasoning:\n\n1. **Context Grounding**: The answer references the document's contents accurately, mentioning specific pet peeves and workplace behavioral expectations.\n2. **Relevance**: The answer is highly relevant to the question, directly addressing the pet peeves listed in the document.\n3. **Conciseness**: Although the answer provides a comprehensive list, it does so without unnecessary elaboration, making it relatively concise.\n4. **Correctness**: The answer closely follows the details provided in the document without deviating into unsupported territory.\n5. **Specificity**: The answer effectively captures the specific examples given in the document, such as harassment, drug use, and unacceptable behavior.\n\nThe answer is well-rounded, accurate, and direct, satisfying all the evaluation criteria effectively.\n\nFinal Result: Good"</li><li>"Reasoning:\nThe given answer diverges considerably from the question and the sources. \n\n1. **Context Grounding**: The information provided in the answer about personal document systems, password managers, secure sharing tools, and feedback processes, while taken from the document, is not contextually grounded in the specific query about accessing training resources.\n\n2. **Relevance**: The answer does not directly address how to access the company's training resources, failing to stay relevant to the primary focus of the question. Instead, it focuses on other topics such as security measures, password management, and feedback processes.\n\n3. **Conciseness**: While the answer is relatively concise, it includes irrelevant procedural details that do not help in answering the specific question about training resources.\n\n4. **Specificity**: The response lacks specific instructions or steps on how one can access the training resources, missing crucial details that might be contained within the document.\n\nAs a result, the provided answer fails to adhere to the established criteria and doesn't fulfill the primary intent of the question.\n\nFinal result: Bad"</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| 1     | <ul><li>'Reasoning:\nThe provided answer is an extensive mishmash of ideas, but evaluating its alignment with the question and the criteria will determine its acceptability:\n\n1. **Context Grounding**: The provided answer is grounded in the tips mentioned in the document. It mentions giving feedback at the time of the event or shortly after, focusing on the situation and not the person, avoiding the blame game, the intention behind feedback, clarity, directness, and appreciation. It also points out key tips for receiving feedback, aligning with the document’s sentiment.\n\n2. **Relevance**: The answer covers the appropriate points relevant to giving and receiving feedback. However, it incorporates more details about receiving feedback rather than sticking strictly to providing feedback as the question specifies.\n\n3. **Conciseness**: The answer could be more concise. There are some repetitive elements and slight tangents, such as unnecessary mentioning of names ("Christopher Estes" and "emichelle James Johnson MD"), which are irrelevant and confusing.\n\n4. **Attempting to Respond if Question is Unanswerable**: The question is answerable based on the document, and the answer appropriately attempts to respond leveraging the document\'s content.\n\n5. **Specifics**: The answer has specifics but these should be more cohesively structured to avoid filler details.\n\n6. **General vs. Specific**: When tips are asked, the answer should not be too general. It includes relevant tips but could further streamline to directly connect with the specific actions as per the document.\n\nTaking the above points into account, here\'s the final decision:\n\nThe response is relevant and includes accurate information, but due to the lack of conciseness, irrelevant name placements, and partial deviation from the primary focus (giving feedback), it is ultimately assessed as:\n\nFinal Result: Bad'</li><li>"Reasoning:\n1. **Context Grounding**: The answer does well in basing its explanation on the provided document's content. The document mentions the importance of sharing information from high-level meetings to provide context, ensure alignment with the organization's vision, mission, and strategy, and foster transparency and collaboration.\n  \n2. **Relevance**: The response addresses the specific question asked, discussing the importance of sharing high-level meeting information and its benefits.\n\n3. **Conciseness**: While the answer is relatively succinct, it slightly over-explains some points. For instance, mentioning the creation of an open and collaborative work environment could be perceived as redundant.\n\n4. **Specificity**: The answer captures more than just the base reason for sharing high-level meeting content, such as addressing concerns and fostering collaboration, but does not detail nuances that may better support the argument (like the decision-making processes mentioned in the document).\n\n5. **Relevance of Tips**: The response includes tips relevant to the question, such as promoting transparency and ensuring team alignment. \n\nThe provided answer effectively captures the primary reasons highlighted in the documents for why it's essential to proactively share information from high-level meetings, despite a small degree of redundancy and slight lack of specific details from the text.\n\nFinal Result: Good"</li><li>'Reasoning:\nThe answer provided attempts to address the question by specifying the need to keep track of kilometers, send an email or excel document to certain email addresses, and provides the reimbursement rate. However, there are several errors and inaccuracies:\n\n1. **Context Grounding**: The answer misrepresents the email addresses, incorrectly formatting them and adding irrelevant names (Dustin Chan.ahouse@example.net). This does not align well with the source document.\n2. **Relevance**: The answer includes relevant steps on how to report car travel expenses, but the incorrect details about email addresses reduce its accuracy and usefulness.\n3. **Conciseness**: The answer is relatively concise but could be better structured for clarity.\n4. **Specificity**: The details given are relatively specific to car travel reimbursement but are flawed due to inaccuracies in email addresses.\n5. **General Accuracy**: The requirement of requesting a parking card and providing the date "2004-04-14" is incorrect per the document.\n\nOverall, the inaccuracies in the email addresses and additional misleading details significantly impact the correctness and reliability of the answer.\n\nFinal result: Bad'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.7015   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_improved-cot-instructions_chat_few_shot_generated_only_re")
# Run inference
preds = model("Reasoning:
1. **Context Grounding**: The answer appropriately uses information from the document to address the query. The email provided (finance@ORGANIZATION_2.<89312988>) matches the reference in Document 1.

2. **Relevance**: The answer is relevant to the question and does not deviate into unrelated topics. It directly addresses the contact point for questions on travel reimbursement.

3. **Conciseness**: The answer is clear and to the point, effectively communicating the necessary information without unnecessary details.

4. **Specificity**: The email address given in the answer is taken directly from the provided document, ensuring the response is specific and accurate.

5. **Overall Completeness**: The answer meets all the criteria mentioned and does not attempt to address unrelated issues or give overly general information.

Final evaluation: Good")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median   | Max |
|:-------------|:----|:---------|:----|
| Word count   | 77  | 162.5692 | 290 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 32                    |
| 1     | 33                    |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (5, 5)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0061 | 1    | 0.244         | -               |
| 0.3067 | 50   | 0.2686        | -               |
| 0.6135 | 100  | 0.2409        | -               |
| 0.9202 | 150  | 0.1401        | -               |
| 1.2270 | 200  | 0.0056        | -               |
| 1.5337 | 250  | 0.0028        | -               |
| 1.8405 | 300  | 0.0022        | -               |
| 2.1472 | 350  | 0.0021        | -               |
| 2.4540 | 400  | 0.0018        | -               |
| 2.7607 | 450  | 0.0016        | -               |
| 3.0675 | 500  | 0.0016        | -               |
| 3.3742 | 550  | 0.0015        | -               |
| 3.6810 | 600  | 0.0014        | -               |
| 3.9877 | 650  | 0.0014        | -               |
| 4.2945 | 700  | 0.0014        | -               |
| 4.6012 | 750  | 0.0014        | -               |
| 4.9080 | 800  | 0.0013        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.0
- Transformers: 4.44.0
- PyTorch: 2.4.1+cu121
- Datasets: 2.19.2
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->