setfit safetensors bert sentence-transformers text-classification generated_from_setfit_trainer arxiv:2209.11055 base_model:BAAI/bge-base-en-v1.5 base_model:finetune:BAAI/bge-base-en-v1.5 model-index region:us
---
base_model: BAAI/bge-base-en-v1.5
library_name: setfit
metrics:
- accuracy
pipeline_tag: text-classification
tags:
- setfit
- sentence-transformers
- text-classification
- generated_from_setfit_trainer
widget:
- text: 'The provided answer is quite disjointed and does not directly address the
    specific question about accessing the company''s training resources. The listed
    methods are more related to accessing various internal tools and processes rather
    than directly answering the question. Let''s break down the problems:


    1. **System and personal documents**: Mentions accessing personal documents, contracts,
    and making reimbursements, which are not directly related to training resources.

    2. **Password Manager (1Password)**: This piece of information about managing
    passwords is irrelevant to accessing training resources.

    3. **Tresorit**: Focuses on secure information sharing, not on training resources.

    4. **Coffee session for feedback**: This is related to clarifying feedback, not
    directly about accessing training resources.

    5. **Learning Budget**: This section is somewhat relevant but is more about requesting
    financial support for training rather than providing a direct method to access
    training resources.


    The answer needs to clearly outline specific steps or platforms dedicated to training
    resources, which is mostly missing here.


    Final Result: ****'
- text: 'The answer succinctly addresses the question by stating that finance@ORGANIZATION_2.<89312988>
    should be contacted for questions about travel reimbursement. This is correctly
    derived from the provided document, which specifies that questions about travel
    costs and reimbursements should be directed to the finance email.


    Final evaluation:'
- text: 'The answer provided correctly mentions the essential aspects outlined in
    the document, such as the importance for team leads to actively consider the possibility
    of team members leaving, to flag these situations to HR, analyze problems, provide
    feedback, and take proactive steps in various issues like underperformance or
    lack of growth. The answer also captures the significance of creating a supportive
    environment, maintaining alignment with the company''s vision and mission, ensuring
    work-life balance, and providing regular feedback and praise.


    However, while the answer is generally comprehensive, it could be slightly more
    direct and concise in its communication. The document points to the necessity
    for team leads to think about potential exits to preemptively address issues and
    essentially prevent situations that may necessitate separation if they aren''t
    managed well. This could have been emphasized more clearly.


    Overall, the answer aligns well with the content and intent of the document.


    Final evaluation:'
- text: 'Reasoning:

    The answer provided is relevant to the question as it directs the user to go to
    the website and check job ads and newsletters for more information about ORGANIZATION.
    However, it lacks comprehensive details. It only partially addresses how one can
    understand ORGANIZATION''s product, challenges, and future since the documents
    suggest accessing job ads and newsletters, and no further content or documents
    were leveraged to provide insights into product details, current challenges, or
    future plans.


    Final evaluation:'
- text: 'Evaluation:

    The answer provides a detailed response directly addressing the question, mentioning
    that the ORGANIZATION_2 and key individuals like Thomas Barnes and Charlotte Herrera
    play a supportive role in the farewell process. This includes handling paperwork,
    providing guidance, and assisting with tough conversations. The answer aligns
    well with the details provided in the document.


    The final evaluation:'
inference: true
model-index:
- name: SetFit with BAAI/bge-base-en-v1.5
  results:
  - task:
      type: text-classification
      name: Text Classification
    dataset:
      name: Unknown
      type: unknown
      split: test
    metrics:
    - type: accuracy
      value: 0.6716417910447762
      name: Accuracy
---

# SetFit with BAAI/bge-base-en-v1.5

This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.

The model has been trained using an efficient few-shot learning technique that involves:

1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
2. Training a classification head with features from the fine-tuned Sentence Transformer.

## Model Details

### Model Description
- **Model Type:** SetFit
- **Sentence Transformer body:** [BAAI/bge-base-en-v1.5](https://huggingface.co/BAAI/bge-base-en-v1.5)
- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
- **Maximum Sequence Length:** 512 tokens
- **Number of Classes:** 2 classes
<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
<!-- - **Language:** Unknown -->
<!-- - **License:** Unknown -->

### Model Sources

- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)

### Model Labels
| Label | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
|:------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1     | <ul><li>'The answer accurately matches the document, confirming that a dime holds a monetary value of 10 cents, which is one-tenth of a dollar. This answer is relevant to the question and appropriately grounded in the provided document.\n\nThe final evaluation:'</li><li>'The answer states that "Father Josh Carrier held the professorship of Chemistry and Physics at Notre Dame." However, the document clearly mentions "Father Joseph Carrier, C.S.C." as holding those positions, not Father Josh Carrier. There is no mention of a Father Josh Carrier in the provided document, making this a case of incorrect naming.\n\nFinal evaluation:'</li><li>'The answer provided is well-aligned with the information from the document. It gives multiple methods for grinding weed without a grinder such as using scissors in a shot glass, a kitchen knife on a cutting board, and a coffee bean grinder. These methods reflect the detailed instructions provided in the document. \n\nFurthermore, it covers several alternative grinding techniques like using scissors in a glass, a pill bottle with a coin, a coffee grinder, a kitchen knife, and a mortar and pestle, which are all mentioned in the document.\n\nOverall, the answer accurately uses the information from the document to provide a comprehensive response to the question.\n\nFinal evaluation:\xa0****'</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
| 0     | <ul><li>"The answer provided effectively responds to the question by listing relevant factors clearly and directly, which are also supported by the content in the provided document. \n\nMy reasoning includes:\n\n1. **Relevance to Document**: The factors mentioned in the answer such as aligning the learning opportunity with personal development goals, roles and responsibilities, and investment value align with the guidance provided in Document 1.\n\n2. **Clarity and Completeness**: The answer is clear, logically structured, and comprehensive. It lists each factor separately, making it easy to understand and follow.\n\n3. **Additional Insights**: The answer also brings in the importance of self-assessment, feedback, and staying informed, which are closely related to effectively determining the alignment of learning opportunities with one's role and goals. This is supported by Document 2, which discusses the importance of checking one's work, job and role, and staying informed.\n\nGiven the direct correlation between the response and the document, along with its clarity and thoroughness, the answer is well-founded and robust.\n\nFinal evaluation:"</li><li>'The answer provided directly addresses the question and correctly lists the specific goals expected for editorial/content team members in their first month. The stated goals are clearly grounded in the text from the provided document. Therefore, the answer is clearly relevant and accurate.\n\n**Final Result: **'</li><li>'The answer provided mentions some of the amenities that were lacking as per the visitor’s experience described in the document, namely the lack of a fridge, air conditioning, towels, soap, and a sufficient number of TV channels. Additionally, the absence of a restaurant was correctly highlighted.\n\nHowever, the statement "all hotels built before 2000 are legally required to have these amenities" is not mentioned in the document and seems to be an unsupported addition by the answer. This detracts from the overall accuracy.\n\nGiven that the answer mostly aligns well with the document but includes an erroneous legal claim not supported by the document, the final evaluation is ‘’.'</li></ul> |

## Evaluation

### Metrics
| Label   | Accuracy |
|:--------|:---------|
| **all** | 0.6716   |

## Uses

### Direct Use for Inference

First install the SetFit library:

```bash
pip install setfit
```

Then you can load this model and run inference.

```python
from setfit import SetFitModel

# Download from the 🤗 Hub
model = SetFitModel.from_pretrained("Netta1994/setfit_baai_newrelic_gpt-4o_cot-few_shot_remove_final_evaluation_e1_larger_train_1727")
# Run inference
preds = model("The answer succinctly addresses the question by stating that finance@ORGANIZATION_2.<89312988> should be contacted for questions about travel reimbursement. This is correctly derived from the provided document, which specifies that questions about travel costs and reimbursements should be directed to the finance email.

Final evaluation:")
```

<!--
### Downstream Use

*List how someone could finetune this model on their own dataset.*
-->

<!--
### Out-of-Scope Use

*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->

<!--
## Bias, Risks and Limitations

*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
-->

<!--
### Recommendations

*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
-->

## Training Details

### Training Set Metrics
| Training set | Min | Median  | Max |
|:-------------|:----|:--------|:----|
| Word count   | 16  | 76.8061 | 301 |

| Label | Training Sample Count |
|:------|:----------------------|
| 0     | 126                   |
| 1     | 137                   |

### Training Hyperparameters
- batch_size: (16, 16)
- num_epochs: (1, 1)
- max_steps: -1
- sampling_strategy: oversampling
- num_iterations: 20
- body_learning_rate: (2e-05, 2e-05)
- head_learning_rate: 2e-05
- loss: CosineSimilarityLoss
- distance_metric: cosine_distance
- margin: 0.25
- end_to_end: False
- use_amp: False
- warmup_proportion: 0.1
- l2_weight: 0.01
- seed: 42
- eval_max_steps: -1
- load_best_model_at_end: False

### Training Results
| Epoch  | Step | Training Loss | Validation Loss |
|:------:|:----:|:-------------:|:---------------:|
| 0.0015 | 1    | 0.2816        | -               |
| 0.0760 | 50   | 0.2483        | -               |
| 0.1520 | 100  | 0.2315        | -               |
| 0.2280 | 150  | 0.1599        | -               |
| 0.3040 | 200  | 0.0965        | -               |
| 0.3799 | 250  | 0.0284        | -               |
| 0.4559 | 300  | 0.006         | -               |
| 0.5319 | 350  | 0.0036        | -               |
| 0.6079 | 400  | 0.0031        | -               |
| 0.6839 | 450  | 0.0026        | -               |
| 0.7599 | 500  | 0.0025        | -               |
| 0.8359 | 550  | 0.0024        | -               |
| 0.9119 | 600  | 0.0023        | -               |
| 0.9878 | 650  | 0.0022        | -               |

### Framework Versions
- Python: 3.10.14
- SetFit: 1.1.0
- Sentence Transformers: 3.1.1
- Transformers: 4.44.0
- PyTorch: 2.4.0+cu121
- Datasets: 3.0.0
- Tokenizers: 0.19.1

## Citation

### BibTeX
```bibtex
@article{https://doi.org/10.48550/arxiv.2209.11055,
    doi = {10.48550/ARXIV.2209.11055},
    url = {https://arxiv.org/abs/2209.11055},
    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
    title = {Efficient Few-Shot Learning Without Prompts},
    publisher = {arXiv},
    year = {2022},
    copyright = {Creative Commons Attribution 4.0 International}
}
```

<!--
## Glossary

*Clearly define terms in order to be accessible across audiences.*
-->

<!--
## Model Card Authors

*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->

<!--
## Model Card Contact

*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->