File size: 5,797 Bytes
3305fda 6509679 3305fda 6509679 3305fda |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
---
license: mit
base_model:
- google/gemma-3-270m
pipeline_tag: text-generation
language:
- en
tags:
- mental-health
- cbt
- therapy
- conversational-ai
- gemma-3
- unsloth
- lora
- psychology
---
# Gemma-3 270M Mental Health Fine-tuned Model
## Model Description
This model is a fine-tuned version of Google's Gemma-3 270M, specifically trained for mental health conversational support using Cognitive Behavioral Therapy (CBT) principles. The model has been trained on 5M+ tokens of high-quality mental health conversational data to provide empathetic, supportive, and therapeutically-informed responses.
**Developed by:** Saurav Kumar Srivastava
## Model Details
- **Base Model:** google/gemma-3-270m
- **Model Size:** 270M parameters
- **Training Data:** 5M+ tokens of CBT-based therapeutic conversations
- **Training Method:** LoRA fine-tuning using Unsloth
- **Quantization:** BF16 GGUF format available
- **License:** MIT
## Training Configuration
The model was fine-tuned using the following specifications:
- **LoRA Rank (r):** 8
- **LoRA Alpha:** 8
- **Target Modules:** All attention and MLP modules
- **Batch Size:** 2 (per device) with 4 gradient accumulation steps
- **Learning Rate:** 2e-4
- **Training Steps:** 30 (optimized for efficiency)
- **Optimizer:** AdamW 8-bit
- **Framework:** Unsloth + TRL SFTTrainer
## Intended Use
### Primary Use Cases
- **Mental Health Support:** Providing empathetic conversations and CBT-based guidance
- **Therapeutic Assistance:** Supporting individuals with anxiety, depression, and stress management
- **Educational Tool:** Teaching CBT techniques and mental health awareness
- **Research:** Studying conversational AI in mental health applications
### Limitations
- **Not a Replacement for Professional Help:** This model should not replace licensed mental health professionals
- **Crisis Situations:** Not suitable for handling severe mental health crises or suicidal ideation
- **General Limitations:** As with all language models, may occasionally generate inappropriate or inaccurate responses
## Usage
### Basic Inference
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")
tokenizer = AutoTokenizer.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")
# Prepare conversation
messages = [{
"role": "user",
"content": [{"type": "text", "text": "I've been feeling really anxious lately about work."}]
}]
# Generate response
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=128,
temperature=1.0,
top_p=0.95,
top_k=64,
do_sample=True
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```
### Recommended Inference Settings
- **Temperature:** 1.0
- **Top-p:** 0.95
- **Top-k:** 64
- **Max New Tokens:** 64-256 (depending on desired response length)
## Training Data
The model was trained on a carefully curated dataset of mental health conversations incorporating:
- CBT-based therapeutic dialogues
- Empathetic response patterns
- Crisis de-escalation techniques
- Mindfulness and coping strategies
- Educational mental health content
**Data Volume:** 5M+ tokens of high-quality conversational data
## Evaluation and Performance
The model demonstrates strong performance in:
- Empathetic response generation
- CBT technique application
- Maintaining therapeutic conversation flow
- Appropriate boundary setting
- Educational content delivery
## Ethical Considerations
### Safety Measures
- Trained to redirect users to professional help when appropriate
- Designed to avoid giving specific medical advice
- Incorporates safety guidelines for mental health conversations
- Includes appropriate disclaimers about professional treatment
### Bias and Fairness
- Efforts made to ensure inclusive and culturally sensitive responses
- Regular evaluation for potential biases in mental health recommendations
- Continuous monitoring for harmful or inappropriate outputs
## Technical Specifications
- **Architecture:** Gemma-3 (Transformer-based)
- **Context Length:** 4000 tokens
- **Precision:** BF16
- **Hardware Requirements:** Compatible with consumer GPUs (4GB+ VRAM recommended)
- **Inference Speed:** Optimized for real-time conversation
## Files and Formats
- **Standard Model:** PyTorch format compatible with Transformers library
- **GGUF Format:** Available for llama.cpp and Ollama integration
- **Quantization:** BF16 precision maintained for quality
## Citation
If you use this model in your research or applications, please cite:
```bibtex
@misc{srivastava2025gemma3mentalhealth,
title={Gemma-3 270M Mental Health Fine-tuned Model},
author={Saurav Kumar Srivastava},
year={2025},
howpublished={\url{https://huggingface.co/Skshackster/gemma3-270m-mental-health-fine-tuned-gguf}},
}
```
## Contact and Support
**Developer:** Saurav Kumar Srivastava
- For questions, issues, or collaboration inquiries, please open an issue in the model repository
## Acknowledgments
- **Google** for the Gemma-3 base model
- **Unsloth** for the efficient fine-tuning framework
- **Mental Health Community** for supporting ethical AI development in therapeutic applications
## Disclaimer
This model is designed for educational and supportive purposes only. It should not be used as a substitute for professional mental health treatment. If you are experiencing a mental health crisis, please contact a licensed mental health professional or emergency services immediately.
--- |