setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'Reasoning:

    1. **Context Grounding**: The answer is well-supported by the provided document.
    It accurately follows the provided instructions.

    2. **Relevance**: The answer directly addresses the question about changing the
    reservation reference from the service page to the booking calendar.

    3. **Conciseness**: The answer is clear and to the point, giving step-by-step
    instructions without unnecessary details.

    4. **Correctness and Detail**: The answer provides the correct steps and detailed
    instructions on how to achieve the task asked in the question.


    Final result:'
- text: 'Reasoning:

    1. **Context Grounding**: The provided answer does not directly relate to the
    documentation provided. The question is about removing the time from the booking
    button, but the answer focuses on blocking off time in a calendar, which is a
    completely different issue.

    2. **Relevance**: The answer does not address the specific question. The question
    asks about hiding time on the booking button, while the provided steps explain
    how to block off time in the calendar.

    3. **Conciseness**: Although the answer is clear and to the point for the steps
    outlined, it is entirely off-topic and doesn''t provide relevant instructions
    for the specific issue.

    4. **Correct Instructions**: The instructions do not match the problem at hand.
    The provided steps are detailed for blocking time, not for removing time from
    the booking button.


    Given this analysis, the answer fails to adhere to any of therequired criteria
    effectively.


    Final Result:'
- text: 'Reasoning:


    1. Context Grounding: The answer is well-supported by the provided document and
    directly relates to the question. It describes the steps to enable calendar scheduling
    and recording as directed in the provided help document.

    2. Relevance: The answer specifically addresses the question of why the <ORGANIZATION>
    business email does not have access to calendar scheduling and recording, explaining
    the necessary steps clearly.

    3. Conciseness: The answer is clear and to the point, providing only the necessary
    instructions without unnecessary information.

    4. Instruction Accuracy: The detailed steps mentioned to verify the domain and
    enable the <ORGANIZATION> app are accurate and correctly instructedas per the
    provided document.


    Final result:'
- text: 'Reasoning:

    1. **Context Grounding**: The answer is well-grounded in the provided document.
    It outlines steps directly referenced in the document, such as the need for a
    Premium plan, connected domain, and the general process of uploading a favicon.

    2. **Relevance**: The answer is relevant to the question posed. It explains the
    process of adding a favicon, which addresses the query about incorporating a favicon
    into search engine results.

    3. **Conciseness**: The response is clear and effectively concise, listing step-by-step
    instructions without unnecessary extraneous details.

    4. **Correct and Detailed Instructions**: The answer accurately conveys the steps
    to add a favicon as per the document, although it omits some minor details such
    as ensuring image resolution and the potential delay in search engine updates.


    Final Result:'
- text: 'Reasoning:

    1. Context Grounding: The answer is well-aligned with the document provided. It
    explains the process based on the steps outlined in the section about adding <ORGANIZATION>
    Bookings.

    2. Relevance: The answer is directly relevant to the question being asked, addressing
    how to make the booking page visible.

    3. Conciseness: The answer is concise and to the point, offering clear step-by-step
    instructions without extraneous information.

    4. Correct and Detailed Instructions: The instructions provided are correct and
    thorough, ensuring that the necessary steps are covered, from upgrading the plan
    to adding the bookings function to the site.


    Final result:'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.6666666666666666
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 0     | <ul><li>'Reasoning: \n\n1. Context Grounding: The provided answer lists several services that Kartz Media & PR offers, many of which are mentioned in the document. However, the document does not mention "personalized space travel public relations for interstellar companies," indicating a potential inaccuracy or embellishment.\n   \n2. Relevance: Most of the services outlined in the answer are relevant and found within the document. The majority align with what is described in the company profile, but the inclusion of a service related to space travel public relations is extraneous and irrelevant.\n   \n3. Conciseness: The answer is relatively concise but incorrect information ("personalized space travel public relations for interstellar companies") detracts from its clarity and precision.\n\nFinal result:'</li><li>"The provided answer does not address the question regarding the careers that graduates from the School of Communications, Media, Arts and Design have achieved. Instead, it gives information about Charles Sheedy's role at the University of Notre Dame, which is unrelated to the question.\n\n**Reasoning:**\n1. **Context Grounding:** The answer provided is well-supported by the document but not relevant to the question asked. The document mentions Charles Sheedy in the context of coeducation at Notre Dame, not about graduates' careers.\n2. **Relevance:** The answer does not answer the specific question about the careers of graduates from the School of Communications, Media, Arts and Design.\n3. **Conciseness:** While the answer is concise, it fails to include any relevant informationabout the requested topic.\n\nFinal Result: ****"</li><li>'Reasoning:\n1. Context Grounding: The answer given does not use any information that relates to horseback riding or arena etiquette. It only discusses measures for enhancing the viewing and playing experience in a darts tournament during the pandemic.\n2. Relevance: The answer does not address the question about following arena etiquette while horseback riding. Instead, it talks about unrelated aspects of a darts tournament.\n3. Conciseness: Despite being concise, the information provided is completely unrelated to the question asked.\n\nFinal result:'</li></ul>                                                        |
| 1     | <ul><li>'Reasoning: \n\n1. **Context Grounding:** The answer is generally in line with the context of the document, which discusses signs of infidelity. Both the document and answer cover behavioral changes, physical changes, communication avoidance, and evidence gathering like checking receipts or tickets.\n \n2. **Relevance:** The response addresses the specific question by listing multiple signs of infidelity, including behaviors and potential evidence to watch for.\n\n3. **Conciseness:** The answer is somewhat clear but could be more concise. It gets a bit repetitive, especially regarding contacting the partner and checking for physical evidence.\n\nFinal result:'</li><li>"Reasoning:\n\n1. Context Grounding: The given answer does not make correct use of the provided document's context and includes random numerical sequences, which are irrelevant and incorrect.\n2. Relevance: While the answer somewhat attempts to address the question by outlining steps related to displaying blog posts, the inclusion of random numbers significantly detracts from focus and accuracy.\n3. Conciseness: The presence of irrelevant numerical sequences and unnecessary repetition hinders conciseness.\n4. Correct and Detailed Instructions: The instructions provided are garbled with non-contextual content, making it difficult to follow and implement accurately.\n\nOverall, the answer fails primarily due to the disruption caused by the irrelevant numerical sequences and the lack of clear instructions rooted in the provided document.\n\nResult:"</li><li>'Reasoning:\n\n1. Context Grounding: The answer is consistent with the document\'s content. It discusses selling gold coins both online and in person, which aligns with the document\'s detailed sections on online sales, local dealers, and auction sites.\n\n2. Relevance: The provided answer specifically answers the question of "How to Sell Gold Coins" by providing relevant methods for selling gold coins online, in person to local dealers, and through auction sites. This matches the guidance provided in the document.\n\n3. Conciseness: The answer is concise and to the point. It avoids unnecessary details and clearly outlines the main methods of selling gold coins, which were detailed extensively in the document.\n\nFinal Result:'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.6667   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_wix_qa_gpt-4o_cot-instructions_remove_final_evaluation_e1_larger_train_17")
# Run inference
preds = model("Reasoning:
1. **Context Grounding**: The answer is well-supported by the provided document. It accurately follows the provided instructions.
2. **Relevance**: The answer directly addresses the question about changing the reservation reference from the service page to the booking calendar.
3. **Conciseness**: The answer is clear and to the point, giving step-by-step instructions without unnecessary details.
4. **Correctness and Detail**: The answer provides the correct steps and detailed instructions on how to achieve the task asked in the question.

Final result:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 33  | 94.8197 | 198 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 117                   |
| 1     | 127                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0016 | 1    | 0.2302        | -               |
| 0.0820 | 50   | 0.2559        | -               |
| 0.1639 | 100  | 0.2525        | -               |
| 0.2459 | 150  | 0.1682        | -               |
| 0.3279 | 200  | 0.0961        | -               |
| 0.4098 | 250  | 0.0504        | -               |
| 0.4918 | 300  | 0.0159        | -               |
| 0.5738 | 350  | 0.0041        | -               |
| 0.6557 | 400  | 0.0027        | -               |
| 0.7377 | 450  | 0.0035        | -               |
| 0.8197 | 500  | 0.0027        | -               |
| 0.9016 | 550  | 0.0023        | -               |
| 0.9836 | 600  | 0.002         | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->