setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning:


    **Context Grounding:**

    - The answer accurately pulls information directly from the provided document,
    including specific changes Haribabu Kommi is making to the storage AM. It lists
    the changes in a manner that seems consistent with the details given in the document.


    **Relevance:**

    - The answer is directly relevant to the question, which asked specifically about
    the changes Haribabu Kommi is making to the storage AM. It enumerates the exact
    modifications and additions that are being incorporated based on the email content.


    **Conciseness:**

    - The answer is concise and to the point, listing only the changes mentioned in
    the supplied email without deviating into unrelated topics or providing extraneous
    information.


    Final Result:'
- text: '**Good**


    **Reasoning:**

    1. **Context Grounding:** The answer "China''s Ning Zhongyan won the gold medal
    in the men''s 1,500m final at the speed skating World Cup" is well-supported by
    the provided document, which explicitly states that Ning Zhongyan won the gold
    medal in the men''s 1,500m final.

    2. **Relevance:** The answer directly addresses the specific question asked, identifying
    the athlete who won the gold medal in the men''s 1,500m final.

    3. **Conciseness:** The answer is clear and to the point, providing only the necessary
    information without any additional, unrelated details.'
- text: 'Reasoning why the answer may be good:

    1. **Context Grounding:** The details in the answer about the sizes of the individual
    and combined portraits are directly pulled from the provided document.

    2. **Relevance:** The answer strictly addresses the question about the available
    sizes for the individual and combined portraits without deviating into unrelated
    topics.

    3. **Conciseness:** The answer is concise, directly providing the requested size
    information without including extraneous details.


    Reasoning why the answer may be bad:

    1. There is no discernible reason why this answer may be bad based on the provided
    criteria. It is well-supported by the document, directly answers thequestion,
    and is concise.


    Final Result: ****'
- text: 'Reasoning why the answer may be good:

    1. **Context Grounding:** The answer accurately lists the components found in
    the provided document, such as comprehension questions, writing exercises, discussion
    questions, an additional reading list, semester and full-year schedules, and a
    bibliography. It also includes details about the organization of the guide into
    units and lessons, which is mentioned in the document.

    2. **Relevance:** The answer specifically addresses the question by identifying
    the components of the British Medieval Student Guide.

    3. **Conciseness:** The answer is relatively to the point, mentioning the main
    components without unnecessary elaboration.


    Reasoning why the answer may be bad:

    1. **Context Grounding:** Although the details are generally correct, some parts
    of the provided description are omitted, such as the note that comprehension question
    answers are in the Teacher''s Guide.

    2. **Relevance:** The initial part about the introductory question "Why read great
    literature?" and some other additional comments are not directly related to the
    components of the Student Guide.

    3. **Conciseness:** The answer could be more concise by excluding repeated and
    unrelated information, focusing only on listing the components directly.


    Final Result: **Bad**


    The answer introduces unnecessary elements that are not related to enumerating
    the components of the guide, and it overlooks some specific details provided in
    the document. Overall, the response is correct but not optimal in addressing the
    specific question concisely.'
- text: '**Reasoning:**


    **Why the answer may be good:**

    1. It lists three names of Members of Congress, which directly responds to the
    question.


    **Why the answer may be bad:**

    1. **Context Grounding:** The provided document specifically names Rep. Danny
    Davis as the third Member of Congress, and the first two were Reps. Keith Ellison
    and Barbara Lee, not Andy Harris, Kyle Evans, or Jessica Smith. This indicates
    that the actual names provided in the answer are incorrect and not grounded in
    the given context.

    2. **Relevance:** The answer is irrelevant because it provides incorrect names,
    which does not address the question accurately.

    3. **Conciseness:** The answer is concise, but since it’s incorrect, its brevity
    doesn''t contribute to its correctness.


    **Final Result:**'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.9066666666666666
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            |
|:------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>"Reasoning why the answer may be good:\n1. **Context Grounding**: The answer is well-supported by the provided document and directly quotes relevant information about Patricia Wallace's roles and responsibilities.\n2. **Relevance**: The answer specifically addresses the question asked, detailing the roles and responsibilities of Patricia Wallace without deviating into unrelated topics.\n3. **Conciseness**: The answer is clear, concise, and focuses on the main points relevant to the question, avoiding unnecessary information.\n\nReasoning why the answer may be bad:\n- There is no significant reason to consider the answer bad based on the given criteria. It comprehensively covers the roles and responsibilities of Patricia Wallace as mentioned in the document.\n\nFinal Result:"</li><li>'### Reasoning:\n**Why the answer may be good:**\n1. **Context Grounding:** The answer is directly taken from the document, which states that a dime is one-tenth of a dollar.\n2. **Relevance:** The answer addresses the specific question asked about the monetary value of a dime.\n3. **Conciseness:** The answer is clear and to the point, providing no more information than necessary.\n\n**Why the answer may be bad:**\n1. **Context Grounding:** The document provides additional context and details about the U.S. dollar system which were not included in the answer. However, these details are not directly necessary to answer the question.\n2. **Relevance:** No deviation or unrelated topics are present in the answer. \n3. **Conciseness:** The answer avoids unnecessary information, maintaining itsclarity and brevity. \n\n### Final Result:\n****'</li><li>'Reasoning why the answer may be good:\n- Context Grounding: The answer refers to symptoms like flu-like signs, which are detailed in the provided document. It also mentions the connection with tampon use, the presence of rashes, and the seriousness of seeking medical help, all of which are discussed in the document.\n- Relevance: The answer addresses the question by listing symptoms and highlighting the importance of recognizing them, which directly corresponds to the question asked.\n- Conciseness: The answer is relatively concise while covering most of the essential details related to recognizing TSS.\n\nReasoning why the answer may be bad:\n- Context Grounding: While the answer does mention flu-like symptoms and the association with tampon use, it lacks specific details like fever and other visible signs mentioned in the document.\n- Relevance: The mention of treatment with antibiotics is somewhat relevant but moves slightly away from the specific focus of how to recognize TSS.\n- Conciseness: The answer could be streamlined further by focusing more on the core question of identifying symptoms rather than mentioning treatment.\n\nFinal Result:'</li></ul> |
| 0     | <ul><li>'**Reasoning:**\n\n**Why the answer may be good:**\n1. **Context Grounding:** The answer does affirm Gregory Johnson as the CEO of Franklin Templeton Investments, which is supported by the provided document.\n2. **Relevance:** The answer directly addresses the question regarding the CEO of Franklin Templeton Investments.\n3. **Conciseness:** The answer is relatively clear and to the point, providing the name of the CEO as requested.\n\n**Why the answer may be bad:**\n1. **Context Grounding:** The statement about Gregory Johnson inheriting the position from his father, Rupert H. Johnson, Sr., is not mentioned in the provided document.\n2. **Relevance:** While the primary answer is correct and relevant, the additional information about the inheritance is not relevant to the specific question asked.\n3. **Conciseness:** The answer includes unnecessary information about the inheritance of the position, which was not part of the question.\n\n**Final result:**'</li><li>'Reasoning why the answer may be good:\n1. The answer is well-supported by the provided document, mentioning key steps in diagnosis and treatment such as taking the cat to the vet, using topical antibiotics and anti-inflammatory medications, completing the full course of treatment, and isolating the infected cat.\n2. It directly addresses the specific question of how to treat conjunctivitis in cats.\n3. The answer is clear and to the point, providing practical advice on treatment.\n\nReasoning why the answer may be bad:\n1. The mention of conjunctivitis in cats often resulting from exposure to a rare type of pollen found only in the Amazon rainforest is not supported by the document. This statement is factually incorrect and detracts from the overall accuracy.\n2. It could be more concise by avoiding unnecessary information and focusing solely on the mostcritical points of treatment.\n\nFinal result:'</li><li>"Reasoning why the answer may be good: \n- The answer correctly identifies the College of Arts and Letters as Notre Dame's first college, founded in 1842, which is directly related to the question asked.\n\nReasoning why the answer may be bad:\n- The answer includes an incorrect and unsupported statement about the curriculum for time travel studies, which is not mentioned in the provided document andis irrelevant to the question.\n\nFinal result:"</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.9067   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_rag_ds_gpt-4o_improved-cot-instructions_two_reasoning_remove_final_evalua")
# Run inference
preds = model("**Good**

**Reasoning:**
1. **Context Grounding:** The answer \"China's Ning Zhongyan won the gold medal in the men's 1,500m final at the speed skating World Cup\" is well-supported by the provided document, which explicitly states that Ning Zhongyan won the gold medal in the men's 1,500m final.
2. **Relevance:** The answer directly addresses the specific question asked, identifying the athlete who won the gold medal in the men's 1,500m final.
3. **Conciseness:** The answer is clear and to the point, providing only the necessary information without any additional, unrelated details.")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median   | Max |
|:-------------|:----|:---------|:----|
| Word count   | 50  | 125.2071 | 274 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 95                    |
| 1     | 103                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0020 | 1    | 0.1499        | -               |
| 0.1010 | 50   | 0.2586        | -               |
| 0.2020 | 100  | 0.2524        | -               |
| 0.3030 | 150  | 0.1409        | -               |
| 0.4040 | 200  | 0.0305        | -               |
| 0.5051 | 250  | 0.015         | -               |
| 0.6061 | 300  | 0.0097        | -               |
| 0.7071 | 350  | 0.0107        | -               |
| 0.8081 | 400  | 0.0054        | -               |
| 0.9091 | 450  | 0.0047        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->