Gemma-3-12B-IT Fine-tuned for Elderly Care (Merged)

This is a fine-tuned version of google/gemma-3-12b-it specialized for elderly care conversations. The model was fine-tuned using LoRA (Low-Rank Adaptation) and then the adapters were merged into the base model for easier deployment and inference.

Model Description

This model aims to be an empathetic, patient, and helpful conversational assistant for elderly individuals. It has been trained on a dataset of high-quality prompt-response pairs relevant to scenarios encountered in elderly care. The fine-tuning focused on:

Companionship: Engaging in friendly, supportive, and warm conversations.
Information Seeking (Non-Medical): Answering general knowledge questions and providing information that can be found via web searches (when integrated into an application).
Assistance Requests: Understanding requests for simple reminders or task-related help.
Storytelling: Generating short, uplifting stories.

Base Model: google/gemma-3-12b-it
Fine-tuning Data: high_quality_data_v1_b20_concurrent_v2.jsonl
Fine-tuning Method: LoRA

r: 16
lora_alpha: 32
lora_dropout: 0.05
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj

Source Adapters (Merged into this Model): gemma3_12b_elderly_care_adapters_v11

Intended Uses & Limitations

Intended Uses:

Conversational Companion: Providing a friendly and engaging chat partner for elderly users to combat loneliness and encourage interaction.
Information Resource: Answering general knowledge questions and, if connected to a search API, providing up-to-date information on weather, news, etc.
Simple Task Assistance: Helping with reminders for medication (as a prompt, not a reliable scheduler) or other daily activities, if integrated into a larger application.
Storytelling: Offering light entertainment through short, positive stories.

Limitations:

No Medical Advice: This model is NOT a substitute for professional medical advice, diagnosis, or treatment. Always consult a qualified healthcare provider for any health concerns.
Potential for Inaccuracies: Like all LLMs, the model can generate incorrect or nonsensical information (hallucinations). Responses should be critically evaluated.
Not for Critical Decisions: Do not rely on this model for making critical decisions regarding health, finance, or safety.
Bias: The model may reflect biases present in its training data (both the base model's pre-training data and the fine-tuning dataset).
Safety: While the fine-tuning data is curated, the model might still generate unexpected or inappropriate content in some edge cases. Robust safety filtering in the application layer is recommended.
Context Window: Limited by max_seq_length: 1024 during fine-tuning. Long conversations might lose earlier context.

How to Use

This is a fully merged model. You can load it directly using AutoModelForCausalLM and AutoTokenizer from the Hugging Face transformers library.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "Heshamproduct/gemma-3-12b-elderly-care-merged-v11"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto", # Handles multi-GPU or CPU if CUDA is not available
    torch_dtype=torch.bfloat16 # Recommended for Gemma models; "auto" also works
)

# Gemma instruction format is important for optimal performance
prompt = "### Instruction:\nTell me a short, happy story about a sunny day.\n\n### Response:\n"

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate response
# Adjust generation parameters as needed
outputs = model.generate(
    **inputs,
    max_new_tokens=250,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1,
    pad_token_id=tokenizer.eos_token_id
)

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
# The output will likely include the prompt, so you might want to clean it:
response_only = generated_text.split("### Response:\n")[-1].strip()
print(response_only)

Training Procedure

The model was fine-tuned from google/gemma-3-12b-it using the SFTTrainer from the trl library and PEFT LoRA. The adapters used for this merged model originated from the directory gemma3_12b_elderly_care_adapters_v11.

Key Training Parameters (for the source adapters):

Base Model: google/gemma-3-12b-it
Fine-tuning Dataset: high_quality_data_v1_b20_concurrent_v2.jsonl
Formatting Function: Custom function to structure data as ### Instruction:\n{prompt}\n\n### Response:\n{response}.
Max Sequence Length: 1024 tokens
LoRA Configuration:
- r: 16
- lora_alpha: 32
- lora_dropout: 0.05
- target_modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
Training Arguments:
- per_device_train_batch_size: 2
- gradient_accumulation_steps: 8 (Effective batch size: 16)
- num_train_epochs: 3.0
- learning_rate: 2e-4
- optim: "paged_adamw_8bit"
- bf16: True
- logging_steps: 10
- save_steps: 100
- warmup_ratio: 0.03
- seed: 42
Quantization (during fine-tuning of adapters):
- load_in_4bit: True
- bnb_4bit_quant_type: "nf4"
- bnb_4bit_compute_dtype: torch.bfloat16
- bnb_4bit_use_double_quant: True

The fine-tuning process that generated the source adapters was logged in a file similar to finetune_gemma3_12b_hq_v24.log (potentially with a different version number if adapters v11 were from an earlier run than the v24 log).

Evaluation Results

Formal quantitative evaluation (e.g., perplexity on a holdout set, standardized benchmarks) has not yet been performed. Qualitative assessment during development showed improved adherence to instructions and persona consistency for elderly care scenarios compared to the base model.

Model Card created by Heshamproduct.
This model was fine-tuned and merged as part of an elderly care chatbot project.
Repository: Heshamproduct/gemma-3-12b-elderly-care-merged-v11

Heshamproduct
/

gemma-3-12b-elderly-care-merged-v11