File size: 5,797 Bytes
3305fda
 
6509679
3305fda
 
6509679
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3305fda
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
---
license: mit
base_model: 
- google/gemma-3-270m
pipeline_tag: text-generation
language:
- en
tags:
- mental-health
- cbt
- therapy
- conversational-ai
- gemma-3
- unsloth
- lora
- psychology
---

# Gemma-3 270M Mental Health Fine-tuned Model

## Model Description

This model is a fine-tuned version of Google's Gemma-3 270M, specifically trained for mental health conversational support using Cognitive Behavioral Therapy (CBT) principles. The model has been trained on 5M+ tokens of high-quality mental health conversational data to provide empathetic, supportive, and therapeutically-informed responses.

**Developed by:** Saurav Kumar Srivastava

## Model Details

- **Base Model:** google/gemma-3-270m
- **Model Size:** 270M parameters
- **Training Data:** 5M+ tokens of CBT-based therapeutic conversations
- **Training Method:** LoRA fine-tuning using Unsloth
- **Quantization:** BF16 GGUF format available
- **License:** MIT

## Training Configuration

The model was fine-tuned using the following specifications:

- **LoRA Rank (r):** 8
- **LoRA Alpha:** 8
- **Target Modules:** All attention and MLP modules
- **Batch Size:** 2 (per device) with 4 gradient accumulation steps
- **Learning Rate:** 2e-4
- **Training Steps:** 30 (optimized for efficiency)
- **Optimizer:** AdamW 8-bit
- **Framework:** Unsloth + TRL SFTTrainer

## Intended Use

### Primary Use Cases
- **Mental Health Support:** Providing empathetic conversations and CBT-based guidance
- **Therapeutic Assistance:** Supporting individuals with anxiety, depression, and stress management
- **Educational Tool:** Teaching CBT techniques and mental health awareness
- **Research:** Studying conversational AI in mental health applications

### Limitations
- **Not a Replacement for Professional Help:** This model should not replace licensed mental health professionals
- **Crisis Situations:** Not suitable for handling severe mental health crises or suicidal ideation
- **General Limitations:** As with all language models, may occasionally generate inappropriate or inaccurate responses

## Usage

### Basic Inference

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")
tokenizer = AutoTokenizer.from_pretrained("Skshackster/gemma3-270m-mental-health-fine-tuned-gguf")

# Prepare conversation
messages = [{
    "role": "user",
    "content": [{"type": "text", "text": "I've been feeling really anxious lately about work."}]
}]

# Generate response
text = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
inputs = tokenizer([text], return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=128,
        temperature=1.0,
        top_p=0.95,
        top_k=64,
        do_sample=True
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### Recommended Inference Settings
- **Temperature:** 1.0
- **Top-p:** 0.95
- **Top-k:** 64
- **Max New Tokens:** 64-256 (depending on desired response length)

## Training Data

The model was trained on a carefully curated dataset of mental health conversations incorporating:
- CBT-based therapeutic dialogues
- Empathetic response patterns
- Crisis de-escalation techniques
- Mindfulness and coping strategies
- Educational mental health content

**Data Volume:** 5M+ tokens of high-quality conversational data

## Evaluation and Performance

The model demonstrates strong performance in:
- Empathetic response generation
- CBT technique application
- Maintaining therapeutic conversation flow
- Appropriate boundary setting
- Educational content delivery

## Ethical Considerations

### Safety Measures
- Trained to redirect users to professional help when appropriate
- Designed to avoid giving specific medical advice
- Incorporates safety guidelines for mental health conversations
- Includes appropriate disclaimers about professional treatment

### Bias and Fairness
- Efforts made to ensure inclusive and culturally sensitive responses
- Regular evaluation for potential biases in mental health recommendations
- Continuous monitoring for harmful or inappropriate outputs

## Technical Specifications

- **Architecture:** Gemma-3 (Transformer-based)
- **Context Length:** 4000 tokens
- **Precision:** BF16
- **Hardware Requirements:** Compatible with consumer GPUs (4GB+ VRAM recommended)
- **Inference Speed:** Optimized for real-time conversation

## Files and Formats

- **Standard Model:** PyTorch format compatible with Transformers library
- **GGUF Format:** Available for llama.cpp and Ollama integration
- **Quantization:** BF16 precision maintained for quality

## Citation

If you use this model in your research or applications, please cite:

```bibtex
@misc{srivastava2025gemma3mentalhealth,
  title={Gemma-3 270M Mental Health Fine-tuned Model},
  author={Saurav Kumar Srivastava},
  year={2025},
  howpublished={\url{https://huggingface.co/Skshackster/gemma3-270m-mental-health-fine-tuned-gguf}},
}
```

## Contact and Support

**Developer:** Saurav Kumar Srivastava
- For questions, issues, or collaboration inquiries, please open an issue in the model repository

## Acknowledgments

- **Google** for the Gemma-3 base model
- **Unsloth** for the efficient fine-tuning framework
- **Mental Health Community** for supporting ethical AI development in therapeutic applications

## Disclaimer

This model is designed for educational and supportive purposes only. It should not be used as a substitute for professional mental health treatment. If you are experiencing a mental health crisis, please contact a licensed mental health professional or emergency services immediately.

---