transformers tensorboard safetensors mistral text-generation trl sft conversational autotrain_compatible text-generation-inference endpoints_compatible 4-bit bitsandbytes region:us
---
library_name: transformers
tags:
- trl
- sft
---

# Model Overview

This model is a LoRA fine-tuned version of "mistralai/Mistral-7B-Instruct-v0.3" using a custom dataset "rayruigalHu/Assisted_Creation" for the task of Script Generation. The fine-tuning process was designed to enhance the model's ability to generate natural and dynamic dialogues that follow certain criterions.

This model is train on GoogleColab T4.

Detail of LoRA configuration, and the trained adapter can be found in "rayruigalHu/Mistral-7B-Instruct-v0.3-peft_20250120_130502".

# Intended Use

Generate conversations between two given characters about a pre-defined topic in json format. The script should contain given number of scenes, and particular lines of diaglouge between two characters. And, the total number of word count should match with the required value.

# How to Use
To use this model, load it via the Hugging Face Transformers and bitsandbyte library:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
                        load_in_4bit=True,
                        bnb_4bit_quant_type="nf4",
                        bnb_4bit_compute_dtype=torch.float16, # torch.bfloat16 # 
                        bnb_4bit_use_double_quant=True,
                    ),
)
```
# Training

## Training Data

"rayruigalHu/Synthesia_Assisted_Creation". We split the dataset into 90% train and 10% val subsets.

## Training Process

GoogleColab with T4 GPU and 15G memory was used to fine-tune this model.

LoRA adapters were applied on all linear modules of the base model. Non-trainable parameters were stored in 4-bit quantized precision, bfloat 16B precision were used during computation.

Small batch size with gradient accumulation, and small eval batch size were used to further reduce memory usage during training.

Early-stopping, dropout, weight decay, together with the small batch size to avoid over-fitting.

The default cross-entropy loss function is used in the training process. Refer to "Training metrics" or "tensorboard_logs.txt" for learning curves on the train and validation set.

# Evaluation

Different LoRA and training hyperparameters were tested, as well as several base-models. The best model were selected based on various evaluation metrics on the validation dataset.

## Customized Metrics
We consider the number of senses, dialouge lines, character names, and word count as customed metrics.

## Other Metrics
BERTScore is used to evaluate the semantic similarity of the generated text.

BLEU, and ROUGE are used to check word-level overlap between the generated text to the provided groundtruth.

Perplexity and Grammar can be computed, to evaluate the model's prediction confidence, and language correctness.

# Results
Compare fine-tuned model with the base model on validation set.

Median error:
| Models        | Trivial error | Line error | Word error |
|---------------|---------------|------------|------------|
| Base model    |      0.0      |     0.0    |   11.5     |
| Fine-tuned    |      0.0      |     0.0    |   7.75     |

Average metric evaluation results:
| Models        | BLEU   | ROUGE1   | ROUGE2   | ROUGEL   | BERTScore_p | BERTScore_r | BERTScore_f1 |
|---------------|--------|----------|----------|----------|-------------|-------------|--------------|
| Base model    | 0.0091 | 0.1472   | 0.0344   | 0.132    | 0.7349      | 0.7353      | 0.7344       |
| Fine-tuned    | 0.0075 | 0.1463   | 0.029    | 0.1253   | 0.7402      | 0.7394      | 0.7393       |