amjad-awad/mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-96K-warmup_steps-75

Mistral 7b instruct

This model is a fine-tuned version of mistral-7b-instruct-v0.2-bnb-4bit on the EngSaf dataset for Automatic Essay Grading.

Robust performance on tasks involving Automatic Essay Grading to give a score and rationale

It achieves the following results on the evaluation set:

• Loss: 1.199
• Score Precision: 0.5654
• Score Recall: 0.5331
• Score F1: 0.5229
• Score Accuracy: 0.53

---

• Rationale Precision: 0.6392
• Rationale Recall: 0.6339
• Rationale F1: 0.6338

Model Details

• Base Model: Mistral 7B: https://arxiv.org/abs/2310.06825
• Fine-tuning Dataset: EngSaf: https://arxiv.org/abs/2407.12818.
• Task: Automatic Essay Grading

Training Data

The model is fine-tuned in the EngSaf dataset, curated for Automatic Essay Grading. EngSaf consists of student responses annotated with

• Questions: Typically short-answer or essay-type.
• Correct Answer: answers provided by teachers.
• Student Answers: Actual responses written by students.
• Output Label: The actual student score.
• Feedback: Explanations justifying the given scores.

Example Usage

Below is an example of how to use the model with the Hugging Face Transformers library:

import torch
from unsloth import FastLanguageModel
from transformers import AutoModelForCausalLM, AutoTokenizer

model, tokenizer = FastLanguageModel.from_pretrained(model_name="amjad-awad/mistral-7b-instruct-v0.2-bnb-4bit-EngSaf-96K-warmup_steps-75",max_seq_length=2048,load_in_4bit=True)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=True,
    random_state=3407,
)

user_content = (
    "Provide both a score and a rationale by evaluating the student's answer strictly within the mark scheme range, "
    "grading based on how well it meets the question's requirements by comparing the student answer to the reference answer.\n"
    "Question: What is photosynthesis?\n"
    "Reference Answer: Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize nutrients from carbon dioxide and water. It generally involves the green pigment chlorophyll and generates oxygen as a by-product.\n"
    "Student Answer: Photosynthesis is how plants make their food using sunlight and carbon dioxide. It also gives off oxygen.\n"
    "Mark Scheme: {'1':'Mentions use of sunlight', '2':'Mentions carbon dioxide and water', '3':'Mentions production of oxygen', '4':'Explains synthesis of nutrients or food', '5':'Mentions chlorophyll or green pigment'}"
)

user = [
    {"role":"system", "content": "You are a grading assistant. Evaluate student answers based on the mark scheme. Respond only in JSON format with keys 'score' (int) and 'rationale' (string)."},
    {"role":"user", "content": user_content},
      ]

inputs = tokenizer.apply_chat_template(user, tokenize=True, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(model.device)
generated_ids = model.generate(**inputs, max_new_tokens=128, temperature=0.2, top_k=5, do_sample=False)[0]
new_generated_ids = generated_ids[inputs["input_ids"].shape[1]:]
generated_text = tokenizer.decode(new_generated_ids, skip_special_tokens=True)

print(generated_text)

Results:

{"score": 5, "rationale": "Your answer is correct. You have accurately described the process of photosynthesis, mentioning the use of sunlight, carbon dioxide, and water, and the production of food and oxygen as by-products. Keep up the good work!"}

Training hyperparameters

The following hyperparameters were used during training:

• per_device_train_batch_size:1
• per_device_eval_batch_size:1
• gradient_accumulation_steps:8
• eval_strategy:"steps"
• save_strategy:"steps"
• eval_steps:10
• logging_dir:"./logs"
• logging_steps:10
• save_total_limit:1
• learning_rate:2e-5
• warmup_steps:75
• weight_decay:0.01
• num_train_epochs:3
• load_best_model_at_end:True
• lr_scheduler_type:"cosine"
• metric_for_best_model:"eval_loss"
• greater_is_better:False

Training results

Step	Training Loss	Validation Loss
10	3.246400	3.287983
20	3.205400	3.181967
30	3.084800	2.996949
40	2.820100	2.741735
50	2.617700	2.459760
60	2.293000	2.150476
70	2.009000	1.858367
80	1.725100	1.532251
90	1.420000	1.372761
100	1.357800	1.307858
110	1.287700	1.275092
120	1.272200	1.243866
130	1.227500	1.224301
140	1.208200	1.204156
150	1.164900	1.201462
160	1.121000	1.197694
170	1.199000	1.197176

Framework versions

• Transformers 4.51.3
• Pytorch 2.7.0
• Datasets 3.6.0
• Unsloth 2025.5.6