setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: "Reasoning:\n\n1. **Context Grounding**: The provided answer does not directly\
    \ reference where to access specific training resources. It rather gives a mixture\
    \ of unrelated information such as using a password manager, secure file sharing,\
    \ personal development budget, etc. These points, while valid, do not directly\
    \ pertain to accessing training resources specifically.\n   \n2. **Relevance**:\
    \ The response deviates from speaking directly about accessing training resources\
    \ and instead covers a broader range of topics, which includes using company systems\
    \ for other purposes, security protocols, and personal development budget discussions.\
    \ These areas do not directly answer the question.\n\n3. **Conciseness**: The\
    \ answer lacks conciseness as it includes quite a bit of unrelated information,\
    \ which makes it unnecessarily lengthy and potentially confusing.\n\n4. **Specificity**:\
    \ The answer fails to be specific about how to access training resources. If the\
    \ document does contain the relevant information, it hasn't been effectively pulled\
    \ out or emphasized in the answer.\n\n5. **No-Response Principle**: If a specific\
    \ question cannot be answered based on the given document, it should be clearly\
    \ stated. Here, it seems the document may not contain a straightforward guide\
    \ to accessing training resources, but this isn't clarified.\n\nFinal Result:\
    \ ****"
- text: '**Reasoning:**


    1. **Context Grounding:** The answer is well-supported by Document 1, which specifies
    that questions about travel reimbursement should be directed to finance@ORGANIZATION_2.

    2. **Relevance:** The answer is directly relevant to the question, which asks
    whom to contact about travel reimbursement.

    3. **Conciseness:** The answer is concise and directly addresses the question
    without including any unnecessary information.

    4. **Response Appropriateness:** The answer correctly utilizes the detail provided
    in the document and does not attempt to provide unnecessary information.

    5. **Specificity:** The answer provides the specific email address for the relevant
    point of contact.

    6. **Balanced Detail:** The answer includes the required contact information and
    is not too general.


    **Final Result: .**'
- text: 'The answer provided fulfills the criteria as outlined:


    1. **Context Grounding**: The answer accurately reflects the contents of the documents,
    particularly Document 1, which specifies the steps team leads should take when
    they consider the possibility that someone''s time at the ORGANIZATION is up.


    2. **Relevance**: The answer directly addresses the question by explaining why
    it is important for team leads to think about the possibility of someone leaving,
    including the potential benefits of addressing issues before they become unmanageable.


    3. **Conciseness**: The answer is somewhat verbose but manages to stay mostly
    on-topic and addresses the question throughout, ensuring it doesn''t veer into
    unrelated topics.


    4. **Specificity**: While a bit generalized in parts, the answer includes specifics
    such as underperformance, lack of growth, and disagreement with company direction,
    which are supported by the details in the documents.


    Therefore, despite minor verbosity, the answer is accurate, relevant, and detailed
    enough to be considered .


    **Final Result: **'
- text: 'Reasoning:

    1. Context Grounding: The response draws from the documents providing relevant
    sources such as the organization''s website, job ads, and newsletter link.

    2. Relevance: The answer is directly related to the question about understanding
    the organization''s products, challenges, and future.

    3. Conciseness: The answer is clear and to the point.

    4. Does not attempt to respond when the document lacks information: It addresses
    the question appropriately with the available information.

    5. Specificity: The answer is specific and provides concrete steps to follow.

    6. Relevant tips: The answer includes actionable steps like visiting the website,
    viewing job ads, and signing up for a newsletter, which are relevant.


    The answer precisely matches all the criteria set for evaluation.


    Final Result:'
- text: 'Reasoning:

    1. Context Grounding: The answer is well-grounded in the document, explaining
    the roles of ORGANIZATION_2, Thomas Barnes, and Charlotte Herrera correctly based
    on the provided information.

    2. Relevance: The answer directly addresses the question, detailing the extent
    of ORGANIZATION_2''s participation in the farewell process.

    3. Conciseness: While mostly concise, the answer could be slightly more succinct.
    Repetition of certain elements could be minimized.

    4. Appropriateness of Response: The answer does not attempt to respond beyond
    the document.

    5. Specificity: The answer is specific, describing the involvement of specific
    people and their roles.

    6. Tips and Generality: The answer includes relevant insights into the roles,
    though it does not go into overly general territory.


    Final Result:'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 1.0
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|:------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>'Reasoning:\n1. **Context Grounding**: The provided answer accurately reflects the information in the document that the Partner Dashboard, including the calendar, cannot currently be accessed via mobile devices and requires supported desktop browsers. This aligns perfectly with the statement in the document.\n2. **Relevance**: The response is directly related to the question about accessing the calendar on mobile devices.\n3. **Conciseness**: The answer is clear and straightforward, avoiding unnecessary details while keeping focused on the main point.\n4. **Correct and Detailed Instructions**: The answer provides correct instructions, including the suggestion to vote for the feature as a form of showing interest, which is detailed in the document.\n\nFinal Result:'</li><li>'Reasoning:\n\n1. Context Grounding: The provided answer accurately reflects the information from the document regarding the 2002 Chrysler Concorde LX for sale in Orlando, FL. The price and mileage data are directly taken from the document.\n2. Relevance: The answer is specifically focused on the price and mileage of the 2002 Chrysler Concorde LX in Orlando, FL, which is precisely what the question asked.\n3. Conciseness: The answer is concise and to the point, providing only the necessary information without any additional, unnecessary details.\n\nFinal Result:'</li><li>'Reasoning:\n1. Context Grounding: The answer needs to be verified against the information in the document. The document lists "DeleteQuery" as an event name but it should be confirmed to ensure accuracy.\n2. Relevance: The answer specifically relates to the value of the key name <SWIFTCODE> when the event name is DeleteQuery.\n3. Conciseness: The answer should provide only the necessary value without additional, unrelated information.\n4. Specificity: The answer should directly specify the value corresponding to <SWIFTCODE> for the event name DeleteQuery.\n5. Key/Value/Event Accuracy: The provided answer needs to match the correct value in the document.\n\nBased on the document:\n- Event name(s) is DeleteQuery\n- Anthonyview key name is <SWIFTCODE>\n- Value is queryName\n\nThe provided answer "queryName" is correct, as it matches the document precisely.\n\nFinal Result:'</li></ul>                                                                                                                                                                                            |
| 0     | <ul><li>'**Reasoning:**\n\n1. **Context Grounding**: The provided document does not mention "Father Josh Carrier" but clearly speaks about "Father Joseph Carrier, C.S.C." and details his roles. Thus, the argument surrounding "Father Josh Carrier" is not properly grounded in the provided context. Additionally, the document does not support the claim that he was a football coach.\n   \n2. **Relevance**: While the answer attempts to address the professorship held, it inaccurately names the individual (Josh instead of Joseph) and adds irrelevant and incorrect information about him being a football coach.\n   \n3. **Conciseness**: The answer introduces unnecessary details by incorrectly stating he was a football coach, which deviates from the core question.\n\n**Final Result: **'</li><li>"Reasoning:\n1. Context Grounding: The answer does not accurately reflect the document's context. The document explains how to filter posts in the blog dashboard for site administrators, not how to add filters for visitors to search blog posts.\n2. Relevance: The answer does not address the visitor experience or how to set up filters for visitors. Instead, it details steps for internal filtering through the dashboard.\n3. Conciseness: While the answer is concise, it is not relevant to the question asked.\n4. Correctness and Detail: The instructions provided do not pertain to setting up filters for visitor searches on the blog, making them incorrect for the question posed.\n\nFinal Result:"</li><li>"Reasoning:\n1. Context Grounding: The answer begins with the process of cutting fishing line, which is aligned with the document, and it also mentions the need for extra line, reflecting the document details. The use of overhand or square knots and stringing from the unknotted end follows the document's instructions. However, the mention of ancient beaders and the full moon is not grounded in the provided document, which is an unnecessary and unrelated addition.\n2. Relevance: The answer predominantly covers the relevant steps for beading, such as cutting the fishing line, stringing beads, and tying knots. However, the historical reference about beaders and the full moon does not address how to bead and detracts from the main task.\n3. Conciseness: The answer is relatively concise but could exclude the historical context that does not contribute to the step-by-step beading process, making it a bit cluttered.\n\nFinal Result:"</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 1.0      |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-instructions_remove_final_evaluation_e1_one_out_17270")
# Run inference
preds = model("Reasoning:
1. Context Grounding: The response draws from the documents providing relevant sources such as the organization's website, job ads, and newsletter link.
2. Relevance: The answer is directly related to the question about understanding the organization's products, challenges, and future.
3. Conciseness: The answer is clear and to the point.
4. Does not attempt to respond when the document lacks information: It addresses the question appropriately with the available information.
5. Specificity: The answer is specific and provides concrete steps to follow.
6. Relevant tips: The answer includes actionable steps like visiting the website, viewing job ads, and signing up for a newsletter, which are relevant.

The answer precisely matches all the criteria set for evaluation.

Final Result:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median   | Max |
|:-------------|:----|:---------|:----|
| Word count   | 32  | 104.4858 | 245 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 381                   |
| 1     | 393                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0005 | 1    | 0.2622        | -               |
| 0.0258 | 50   | 0.2676        | -               |
| 0.0517 | 100  | 0.2524        | -               |
| 0.0775 | 150  | 0.2505        | -               |
| 0.1034 | 200  | 0.253         | -               |
| 0.1292 | 250  | 0.2458        | -               |
| 0.1550 | 300  | 0.2191        | -               |
| 0.1809 | 350  | 0.1842        | -               |
| 0.2067 | 400  | 0.1665        | -               |
| 0.2326 | 450  | 0.1262        | -               |
| 0.2584 | 500  | 0.093         | -               |
| 0.2842 | 550  | 0.0614        | -               |
| 0.3101 | 600  | 0.0524        | -               |
| 0.3359 | 650  | 0.0346        | -               |
| 0.3618 | 700  | 0.0412        | -               |
| 0.3876 | 750  | 0.0246        | -               |
| 0.4134 | 800  | 0.0183        | -               |
| 0.4393 | 850  | 0.0165        | -               |
| 0.4651 | 900  | 0.0193        | -               |
| 0.4910 | 950  | 0.0134        | -               |
| 0.5168 | 1000 | 0.0044        | -               |
| 0.5426 | 1050 | 0.0097        | -               |
| 0.5685 | 1100 | 0.0085        | -               |
| 0.5943 | 1150 | 0.0088        | -               |
| 0.6202 | 1200 | 0.0079        | -               |
| 0.6460 | 1250 | 0.0042        | -               |
| 0.6718 | 1300 | 0.003         | -               |
| 0.6977 | 1350 | 0.0038        | -               |
| 0.7235 | 1400 | 0.0072        | -               |
| 0.7494 | 1450 | 0.0017        | -               |
| 0.7752 | 1500 | 0.0024        | -               |
| 0.8010 | 1550 | 0.0019        | -               |
| 0.8269 | 1600 | 0.0015        | -               |
| 0.8527 | 1650 | 0.0015        | -               |
| 0.8786 | 1700 | 0.0014        | -               |
| 0.9044 | 1750 | 0.0014        | -               |
| 0.9302 | 1800 | 0.0014        | -               |
| 0.9561 | 1850 | 0.0014        | -               |
| 0.9819 | 1900 | 0.0013        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->