peft safetensors text-classification en base_model:FacebookAI/roberta-base base_model:adapter:FacebookAI/roberta-base region:us
---
base_model: roberta-base
language:
- en
library_name: peft
metrics:
- accuracy
pipeline_tag: text-classification
---

# Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->
This model classifies whether the text/email is spam or ham. 

## Model Details
This model is a fine-tuned version of roberta-base using LoRA specifically for a binary classification task containing 
emails (subject + message) and a label whether it is spam or ham. 

### Model Description

<!-- Provide a longer summary of what this model is. -->
The base model for this fine-tuning is roberta-base, which is a transformer-based model pre-trained on a large corpus of English data in a self-supervised fashion. 
RoBERTa is an optimized version of BERT, designed to perform better on natural language understanding tasks.
We have applied LoRA to adapt the original RoBERTa model to the specific nuances of our binary classification problem. 
LoRA introduces low-rank matrices that are trained during the fine-tuning process, enabling the model to learn task-specific 
adaptations without altering the pre-trained weights directly.

- **Developed by:** Likhith231
- **Model type:** Text Classification
- **Finetuned from model:** Roberta Base

### Training Data
The model was finetuned on truncated version of SetFit/enron_spam dataset. The SetFit/enron_spam consists of 33716 rows. 
The truncated version consists of 1000 train samples and 1000 test samples with columns, text and label.


### Model Sources

<!-- Provide the basic links for the model. -->

- **Repository:** https://huggingface.co/FacebookAI/roberta-base

## Training Details

- pretrained model= Roberta-base
- all params =  125,313,028
- trainable params=  665,858
- trainable% = 0.531355766137899

### Parameters
- weight_decay = 0.01
- lr = 1e-3
- batch_size = 4
- num_epochs = 10

### Results

Epoch|Training Loss|Validation Loss|Accuracy
-----|-------------|---------------|---------
1    |No log       |0.172788       |0.957
2    |0.194500     |0.202991       |0.956
3    |0.194500     |0.229950       |0.958
4    |0.038400     |0.267390       |0.954
5    |0.038400     |0.283116       |0.963
6    |0.007000     |0.254960       |0.961
7    |0.007000     |0.299375       |0.961
8    |0.007900     |0.276321       |0.966
9    |0.007900     |0.275304       |0.967
10   |0.002000     |0.271234       |0.967



### Framework versions

- PEFT 0.8.2