PRS-Gemma3-1B

This model is a fine-tuned version of google/gemma-3-1b-it using LoRA (Low-Rank Adaptation).

Model Details

• Base Model: google/gemma-3-1b-it
• Fine-tuning Method: LoRA
• Training Epochs: 5
• LoRA Rank: 16
• LoRA Alpha: 32
• Learning Rate: 5e-4
• Max Length: 580 tokens
• Training Samples: 4950
• Validation Samples: 550
• Attention Implementation: eager (optimized for Gemma3)

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it")
base_model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-3-1b-it",
    attn_implementation="eager"
)
model = PeftModel.from_pretrained(base_model, "singhprabhat/PRS-Gemma3-1B")

# Generate text
input_text = "### Instruction:\nYour instruction here\n\n### Response:\n"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=512, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Data

The model was fine-tuned on instruction-output pairs with a 90/10 train-validation split. Training was performed with evaluation every 250 steps using eager attention implementation for optimal Gemma3 performance.