setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning: The given answer is well-supported by the provided document and
    includes details that are directly relevant to the identification of a funnel
    spider, such as the dark brown or black body and the presence of a hard, shiny
    carapace. It also mentions the two large fangs which is a key characteristic.
    The answer stays focused on the specific question of identifying a funnel spider,
    without deviating into unrelated topics.


    Final Result:'
- text: 'Reasoning:

    1. **Context Grounding**: The provided answer aligns well with the document''s
    instructions for writing in MLA format, covering key points such as margins, font
    size, line spacing, and the structure of the heading and title.

    2. **Relevance**: It directly addresses the question by focusing on the MLA formatting
    guidelines, without introducing unrelated information.

    3. **Conciseness**: The answer is succinct and stays concentrated on the main
    steps required to format a paper according to MLA rules, without unnecessary elaboration.


    Final Result:'
- text: "Reasoning: \n\n1. **Context Grounding**: The answer provided aligns with\
    \ the document stating the importance of Biology, Chemistry, Physics, and Mathematics\
    \ (BCPM), clinical experience, and preparing for the MCAT. However, it does not\
    \ mention the psychological and sociological prerequisites for the MCAT and disregards\
    \ some essential steps discussed in the document such as obtaining letters of\
    \ evaluation, having a well-rounded college experience, or demonstrating leadership\
    \ skills.\n   \n2. **Relevance**: The answer does address the overall steps to\
    \ get into medical school but oversimplifies the process. It neglects other critical\
    \ factors from the document such as diversified majors, the importance of guidance\
    \ from professors, and gaining hands-on experience through internships and involvement\
    \ in community health settings.\n\n3. **Conciseness**: The answer is succinct\
    \ and to the point, which aids in clarity but fails to encompass the comprehensive\
    \ approach required outlined in the document.\n\nFinal Result:"
- text: 'Reasoning:

    1. Context Grounding: The answer is not fully grounded in the provided document.
    While some elements align with the document such as hiding in the same room after
    the seeker leaves and using creative hiding strategies, some examples like hiding
    behind a bright red couch and hovering in a nearby corner are not directly supported
    by the provided text.

    2. Relevance: The answer somewhat addresses the specific question of how to be
    a master at hide and seek, but it introduces elements that were not explicitly
    mentioned in the document.

    3. Conciseness: Although the answer is relatively concise, some parts are not
    necessary and detract from the main focus.


    Final Result:'
- text: 'Reasoning:

    1. Context Grounding: The provided answer details steps to use a saline solution
    for treating a baby''s cough. However, it does not accurately reflect the ratio
    and method provided in the document.

    2. Relevance: The answer addresses the question of how to treat a baby''s cough
    but narrowly focuses on only one treatment method, despite the document describing
    other methods.

    3. Conciseness: The answer contains a fair amount of detail, but it introduces
    additional unnecessary procedures not mentioned in the document (e.g., tip of
    the suction cup an inch into the nostrils).


    Final Result:'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.8108108108108109
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|:------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>"Reasoning: \n\n1. Context Grounding: The provided document details the events leading to Joan Gaspart’s resignation and confirms it happened after a poor season in 2003.\n2. Relevance: The answer directly addresses the specific question about who resigned from the presidency after Barcelona's poor showing in the 2003 season.\n3. Conciseness: The answer is clear and to the point without unnecessary information.\n\nFinal Result:"</li><li>"Reasoning:\n1. **Context Grounding**: The answer appropriately pulls references from the document, mentioning the hazards of working with electricity and the potential for long-term issues if electrical work isn't done correctly, which aligns with the provided content.\n2. **Relevance**: The answer directly addresses why it is beneficial to hire a professional electrician like O’Hara Electric, explicitly tying into the concerns of safety, expertise, and ensuring the job is done correctly on the first attempt.\n3. **Conciseness**: The answer is concise and to the point, avoiding unnecessary information and sticking closely to the reasons why hiring a professional is recommended.\n\nFinal Result:"</li><li>'Reasoning:\n1. Context Grounding: The provided document explicitly states that Aerosmith\'s 1987 comeback album was "Permanent Vacation". The answer is directly supported by this information.\n2. Relevance: The answer is directly related to and completely addresses the question about the title of Aerosmith\'s 1987 comeback album.\n3. Conciseness: The answer is concise and to the point, providing only the necessary information without any extraneous details.\n\nFinal result:'</li></ul>                                                                                                                                                                      |
| 0     | <ul><li>'Reasoning:\n1. **Context Grounding**: The response effectively uses the provided document to form its recommendations. It pulls together various tips on identifying and avoiding triggers, utilizing sensory substitutes, and participating in alternative activities to manage cravings. \n2. **Relevance**: The answer remains focused on the question, directly addressing how to stop cravings for smoking by providing actionable advice and methods.\n3. **Conciseness**: While informative, the answer could benefit from being slightly more concise. Some points, such as the detailed explanation about licorice root,might be trimmed for brevity.\n\nFinal Result:'</li><li>'Reasoning:\n1. **Context Grounding:** The provided answer is rooted in the document, which mentions that Amy Bloom finds starting a project hard and having to clear mental space, recalibrate, and become less involved in her everyday life.\n2. **Relevance:** The response accurately focuses on the challenges Bloom faces when starting a significant writing project, without deviating into irrelevant areas.\n3. **Conciseness:** The answer effectively summarizes the relevant information from the document, staying clear and to the point while avoiding unnecessary detail.\n\nFinal Result:'</li><li>'Reasoning:\n1. Context Grounding: The provided document does have a listing for a 6 bedroom detached house. The factual details in the answer given by the user are not present in the document.\n2. Relevance: The user’s answer lists a different price (£2,850,000 vs. £950,000) and an incorrect address (Highgate Lane, Leeds, Berkshire, RG12 vs Willow Drive, Twyford, Reading, Berkshire, RG10) than the provided document.\n3. Conciseness: The user’s answer, though concise, is factually incorrect based on the document.\n\nFinal Result:'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.8108   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wikisum_gpt-4o_cot-instructions_remove_final_evaluation_e1_larger_train_1")
# Run inference
preds = model("Reasoning: The given answer is well-supported by the provided document and includes details that are directly relevant to the identification of a funnel spider, such as the dark brown or black body and the presence of a hard, shiny carapace. It also mentions the two large fangs which is a key characteristic. The answer stays focused on the specific question of identifying a funnel spider, without deviating into unrelated topics.

Final Result:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 33  | 88.6482 | 198 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 95                    |
| 1     | 104                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0020 | 1    | 0.2432        | -               |
| 0.1004 | 50   | 0.256         | -               |
| 0.2008 | 100  | 0.2208        | -               |
| 0.3012 | 150  | 0.0894        | -               |
| 0.4016 | 200  | 0.0315        | -               |
| 0.5020 | 250  | 0.0065        | -               |
| 0.6024 | 300  | 0.0025        | -               |
| 0.7028 | 350  | 0.0022        | -               |
| 0.8032 | 400  | 0.002         | -               |
| 0.9036 | 450  | 0.002         | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->