setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'The answer provided concisely lists the changes being made to the storage
    AM as per Haribabu Kommi''s email. It addresses the specific question asked and
    is directly related to the content of the provided document. There is no deviation
    into unrelated topics, and unnecessary information is avoided.


    Final Evaluation:'
- text: 'Reasoning:

    1. Context Grounding: The provided document clearly states that China''s Ning
    Zhongyan won the gold medal in the men''s 1,500m final at the speed skating World
    Cup.

    2. Relevance: The answer directly addresses who won the gold medal in the men''s
    1,500m final at the speed skating World Cup, which is the specific question asked.

    3. Conciseness: The answer is short and to the point, providing just the necessary
    information without any unnecessary elaboration.


    Final result:'
- text: 'Evaluation:


    1. **Context Grounding:** The answer correctly states the sizes for both individual
    and combined portraits as listed in the provided document.

    2. **Relevance:** The answer directly addresses the specific question regarding
    the sizes of the portraits available.

    3. **Conciseness:** The answer is clear, to the point, and avoids unnecessary
    information, making it concise.


    Final result:'
- text: "The provided answer is well-supported by the document and directly related\
    \ to the question. It enumerates the components of the British Medieval Student\
    \ Guide accurately. \n\nReasoning:\n1. Context Grounding: The answer is consistent\
    \ with the provided document, which lists the elements of the Student Guide.\n\
    2. Relevance: The answer directly addresses the question asked, providing detailed\
    \ components of the guide.\n3. Conciseness: The answer is clear and to the point,\
    \ avoiding unnecessary information that doesn't pertain to the question.\n\nFinal\
    \ Evaluation:"
- text: 'The answer lists Rep. Andy Harris, Reps. Kyle Evans, and Jessica Smith as
    the first three Members of Congress to call for an end to the blockade of Gaza.
    However, according to the document, the correct individuals are Reps. Keith Ellison,
    Barbara Lee, and Danny Davis. Therefore, the answer is not grounded in the provided
    document and is factually incorrect.


    The final evaluation:'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.9066666666666666
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     |
|:------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>"The answer provided is comprehensive and directly addresses the question. Here is the reasoning:\n\n1. **Context Grounding:** The answer precisely matches the details provided in the document. Patricia Wallace's various roles, including managing a clothing closet, overseeing a food pantry, coordinating the food backpack program, and leading the Intervention Support Team, are well-supported by the text.\n   \n2. **Relevance:** The answer is entirely relevant to the question, as it lists the specific roles and responsibilities of Patricia Wallace at Oak View Elementary as outlined in the document.\n\n3. **Conciseness:** The answer is clear and focused, listing the relevant roles and responsibilities without unnecessaryinformation.\n\nTherefore, the evaluation is:"</li><li>"Reasoning:\n1. Context Grounding: The answer is well supported by the document provided. It details the necessary steps to administer a saline solution to a baby, which is a method found within the source text.\n2. Relevance: The answer focuses specifically on treating a baby's cough, directly addressing the question asked.\n3. Conciseness: The answer is clear and to the point, providing concrete steps without unnecessary information. However, the answer could have been made even more concise by avoiding repetitions about the saline solution preparation.\n\nFinal evaluation:"</li><li>'Reasoning:\n1. Context Grounding: The answer provided accurately reflects the information in the document, describing the symptoms, risk factors, and necessary actions if toxic shock syndrome (TSS) is suspected.\n2. Relevance: The answer directly addresses the question asked, focusing on how to recognize TSS and what to do if you suspect you have it.\n3. Conciseness: The answer effectively condenses the necessary information into a coherent, straightforward explanation without extraneous details.\n\nFinal Evaluation:'</li></ul> |
| 0     | <ul><li>'Evaluation:\nThe answer provided incorrectly identifies the creation of a "literature hall" instead of a "science hall" as mentioned in the document. The answer also correctly attributes the oversight to Fr. Zahm, but this information is related to the wrong type of hall as per the document.\n\n1. Context Grounding: The document specifically states that a "Science Hall" was built under the direction of Fr. Zahm in 1883, not a literature hall.\n2. Relevance: The answer partially addresses the question correctly by mentioning Fr. Zahm, but it misidentifies the type of hall constructed.\n3. Conciseness: The answer is concise but includesincorrect information.\n\nThe final evaluation:'</li><li>'Reasoning:\n1. Context Grounding: The document supports that Gregory Johnson is the CEO of Franklin Templeton Investments and provides sufficient context about his role and relation to the company.\n2. Relevance: The answer directly addresses the question about the CEO of Franklin Templeton Investments.\n3. Conciseness: The answer presents the information clearly and succinctly without unnecessary details.\n\nFinal Result: ****'</li><li>'The answer correctly identifies that retired priests and brothers live at Fatima House. However, the additional information about the rare collection of ancient religious manuscripts at Fatima House is not supported by the document, making it an irrelevant addition. This deviates from the principle of conciseness and relevance to the specific question asked.\n\nFinal evaluation:'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                      |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.9067   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_rag_ds_gpt-4o_cot-few_shot-instructions_remove_final_evaluation_e1_larger")
# Run inference
preds = model("The answer provided concisely lists the changes being made to the storage AM as per Haribabu Kommi's email. It addresses the specific question asked and is directly related to the content of the provided document. There is no deviation into unrelated topics, and unnecessary information is avoided.

Final Evaluation:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 11  | 76.2020 | 196 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 94                    |
| 1     | 104                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0020 | 1    | 0.2119        | -               |
| 0.1010 | 50   | 0.255         | -               |
| 0.2020 | 100  | 0.1703        | -               |
| 0.3030 | 150  | 0.0611        | -               |
| 0.4040 | 200  | 0.0351        | -               |
| 0.5051 | 250  | 0.0197        | -               |
| 0.6061 | 300  | 0.0171        | -               |
| 0.7071 | 350  | 0.0109        | -               |
| 0.8081 | 400  | 0.0109        | -               |
| 0.9091 | 450  | 0.0072        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->