setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning:

    The answer provides a solid overview of identifying a funnel spider, including
    its dark brown or black body, shiny carapace, and large fangs. These points align
    well with the details in the provided document. However, while the answer includes
    the key features described in the document, it misses a few additional characteristics
    such as the spinnerets, size variations, and geographical habitat that are valuable
    in identifying funnel spiders more comprehensively. Nonetheless, the answer remains
    relevant and concise based on the essential points covered.

    Evaluation: Good'
- text: 'Reasoning:

    The answer provides a comprehensive and accurate description of how to write a
    paper in MLA format. It mentions key points such as setting up 1-inch margins,
    using 12-point font, and double-spacing the text, which are directly aligned with
    the instructions in the provided document. It also addresses creating a running
    header, typing the heading information in the upper left corner, and centering
    the paper''s title, all of which are specified in the document. The answer is
    well-supported by the document, relevant to the question asked, and presented
    concisely.


    Final Evaluation: Good

    Evaluation: Good'
- text: 'Reasoning:

    The provided answer offers relevant and practical advice for getting into medical
    school, including focusing on core science subjects, gaining clinical experience,
    and preparing for the MCAT. These points align well with the suggestions in the
    document. However, the answer overlooks several important details such as engaging
    in extracurricular activities, seeking leadership opportunities, and preparing
    a comprehensive application, which the document emphasizes. Including these aspects
    would have made the response more comprehensive and better aligned with the provided
    document.


    Evaluation: Good'
- text: 'Reasoning:

    The provided answer offers several strategic tips for becoming adept at hide and
    seek. The suggestions include creative hiding strategies like staying in the room
    where the seeker started, using camouflage, and hiding in plain sight, which align
    well with the document''s advice. The document recommends looking for long edges
    to hide behind, using dense curtains, hiding in laundry baskets, and looking for
    multi-colored areas, all of which are echoed in the answer. The advice given is
    concise, relevant, and matches the tips and guidelines from the document.


    Final Evaluation: good

    Evaluation: Good


    '
- text: 'Reasoning:

    1. **Context Grounding**: The answer refers to making a saline solution for treating
    a baby''s cough, using a method that significantly deviates from the details provided
    in the document. The document provides specific instructions for a saline solution
    and its administration, but the quantities and steps in the answer do not align
    with the document''s instructions. Additionally, the document does not suggest
    using 2 cups of water and the method of inserting the suction bulb into the baby''s
    nostril about an inch is incorrect and potentially harmful, deviating significantly
    from the recommended depth.

    2. **Relevance**: While the answer aims to address how to treat a baby''s cough,
    it includes incorrect measurements and method details, which makes it less reliable
    and potentially unsafe.

    3. **Conciseness**: The answer is fairly concise but includes inaccuracies that
    make it untrustworthy.


    Due to these points, the answer does not meet the necessary criteria of being
    well-grounded in the document, relevant, or correctly concise.


    Final Evaluation: Bad'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.8666666666666667
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|:------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0     | <ul><li>"Reasoning:\nThe answer provided closely aligns with the specific instructions given in the document about petting a bearded dragon. It correctly mentions using 1 or 2 fingers to gently stroke the dragon's head, lowering your hand slowly to avoid startling it, and washing hands before and after petting to reduce the risk of bacteria transfer. However, the part about using a specific perfume or scent to help the dragon recognize you is not supported by the text and is, in fact, incorrect.\n\nFinal Evaluation: Bad\nResult: Bad\n"</li><li>'Reasoning:\nThe answer provided addresses the physical characteristics of a funnel spider but includes several inaccuracies and deviations from the information in the provided document. Key errors include describing the funnel spider as light brown or gray with a soft, dull carapace, which contradicts the document’s description of a dark brown or black body and a hard, shiny carapace. Additionally, the claim that funnel spiders have 3 non-poisonous fangs pointing sideways is incorrect based on the document, which states that the funnel spider has two large, downward-pointing fangs that are poisonous. The document provides clear and detailed descriptions that should form the basis for an accurate answer.\n\nFinal Evaluation: Bad'</li><li>'Reasoning:\nThe answer provided, "Luis Figo left Barcelona to join Real Madrid," while factually correct according to the provided document, is entirely unrelated to the question "How to Calculate Real Estate Commissions." The document and the answer focus on a historical event in soccer rather than providing any information or calculations related to real estate commissions. \n\nFinal Evaluation: Bad'</li></ul> |
| 1     | <ul><li>'Reasoning:\nThe answer is well-supported by the document and directly relates to the question of how to hold a note while singing. It addresses key aspects such as breathing techniques, posture, and controlled release of air, all of which are mentioned in the provided document. The answer stays concise and clear, without deviating into unrelated topics, effectively summarizing the necessary steps for holding a note.\n\nFinal result: Good'</li><li>'Reasoning:\nThe answer is well-founded in the provided document and directly relates to the question of how to stop feeling empty. It suggests practical actions like keeping a journal, trying new activities, and making new friends, all of which are discussed in the document. The recommendations in the answer are summarized clearly and are appropriate responses to the question without providing extraneous information.\n\nFinal Evaluation:\nGood'</li><li>'Reasoning:\nThe answer aligns well with the instructions provided in the document and effectively addresses the question of how to dry curly hair. It begins by recommending gently squeezing out excess water, followed by the application of a leave-in conditioner and the use of a wide-tooth comb for detangling, which are all steps mentioned in the document. The answer then advises adding styling products and parting the hair to lift the roots, which helps expedite the air-drying process. The key points from the document are reflected in the answer, ensuring it is contextually grounded and relevant.\n\nEvaluation: Good'</li></ul>                                                                                                                                                                  |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.8667   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wikisum_gpt-4o_improved-cot-instructions_chat_few_shot_generated_only_rea")
# Run inference
preds = model("Reasoning:
The answer provides a solid overview of identifying a funnel spider, including its dark brown or black body, shiny carapace, and large fangs. These points align well with the details in the provided document. However, while the answer includes the key features described in the document, it misses a few additional characteristics such as the spinnerets, size variations, and geographical habitat that are valuable in identifying funnel spiders more comprehensively. Nonetheless, the answer remains relevant and concise based on the essential points covered.
Evaluation: Good")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 58  | 93.6479 | 177 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 34                    |
| 1     | 37                    |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (5, 5)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0056 | 1    | 0.2213        | -               |
| 0.2809 | 50   | 0.2497        | -               |
| 0.5618 | 100  | 0.1439        | -               |
| 0.8427 | 150  | 0.0107        | -               |
| 1.1236 | 200  | 0.003         | -               |
| 1.4045 | 250  | 0.0022        | -               |
| 1.6854 | 300  | 0.002         | -               |
| 1.9663 | 350  | 0.0018        | -               |
| 2.2472 | 400  | 0.0017        | -               |
| 2.5281 | 450  | 0.0015        | -               |
| 2.8090 | 500  | 0.0015        | -               |
| 3.0899 | 550  | 0.0014        | -               |
| 3.3708 | 600  | 0.0014        | -               |
| 3.6517 | 650  | 0.0012        | -               |
| 3.9326 | 700  | 0.0013        | -               |
| 4.2135 | 750  | 0.0012        | -               |
| 4.4944 | 800  | 0.0013        | -               |
| 4.7753 | 850  | 0.0012        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.0
- Transformers: 4.44.0
- PyTorch: 2.4.1+cu121
- Datasets: 2.19.2
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->