zoeythanayot/gemma-1b-thai-lora-CoT

Gemma-1B Thai Medical CoT

Model Description

This model is a fine-tuned version of google/gemma-3-1b-it on Thai medical reasoning dataset. It has been optimized to provide Chain-of-Thought (CoT) reasoning for medical questions in Thai language.

Model Details

• Base Model: google/gemma-3-1b-it
• Model Size: 1B parameters
• Language: Thai
• Domain: Medical
• Training Method: LoRA (Low-Rank Adaptation)
• Dataset: medical-o1-reasoning-SFT-TH (22,848 samples)

Training Details

Training Data

• Dataset: RJTPP/medical-o1-reasoning-SFT-TH
• Format: Question → Complex Chain-of-Thought → Response
• Language: Thai
• Domain: Medical and healthcare

Training Configuration

• Method: LoRA fine-tuning
• LoRA Rank: 32
• LoRA Alpha: 64
• Learning Rate: 3e-4
• Batch Size: 1 (with gradient accumulation)
• Optimizer: paged_adamw_8bit
• Training Steps: 1000
• Scheduler: Cosine with warmup

Training Infrastructure

• GPU: L4 (Google Colab)
• Memory Optimization:
  • Gradient checkpointing
  • 4-bit quantization
  • LoRA adapter training

Usage

Loading the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model
base_model_id = "google/gemma-3-1b-it"
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    device_map="auto",
    torch_dtype=torch.float16
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "zoeythanayot/gemma-1b-thai-lora-CoT")

# Merge adapter weights (optional)
model = model.merge_and_unload()

Inference Example

# Example medical question in Thai
input_text = """Question: โรคเบาหวานคืออะไร?
Complex_CoT: โรคเบาหวาน (Diabetes mellitus) เป็นกลุ่มของโรคเมตาบอลิกที่มีลักษณะเฉพาะคือภาวะน้ำตาลในเลือดสูง (hyperglycemia) ซึ่งเกิดจากความผิดปกติของการหลั่งอินซูลิน (insulin secretion), การทำงานของอินซูลิน (insulin action) หรือทั้งสองอย่าง
Response: """

# Tokenize input
input_ids = tokenizer(input_text, return_tensors="pt").to(model.device)

# Generate response
with torch.no_grad():
    outputs = model.generate(
        **input_ids,
        max_new_tokens=256,
        do_sample=True,
        temperature=0.7,
        pad_token_id=tokenizer.eos_token_id
    )

# Decode output
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Model Performance

The model demonstrates strong performance in:

• Thai Medical Question Answering: Provides accurate medical information in Thai
• Chain-of-Thought Reasoning: Shows step-by-step medical reasoning process
• Domain Knowledge: Covers various medical topics and conditions
• Language Quality: Maintains natural Thai language flow

Use Cases

• Medical education and training in Thai
• Healthcare chatbots and virtual assistants
• Medical information retrieval systems
• Clinical decision support tools
• Medical research and documentation

Limitations

• Language: Optimized for Thai language only
• Domain: Focused on medical domain
• Model Size: 1B parameters may limit complex reasoning compared to larger models
• Training Data: Performance depends on the quality and coverage of training data
• Medical Advice: This model is for educational purposes only and should not replace professional medical consultation

Ethical Considerations

• Medical Accuracy: While trained on medical data, outputs should be verified by medical professionals
• Liability: Users are responsible for validating medical information before application
• Professional Consultation: This model does not replace qualified medical advice
• Bias: May reflect biases present in training data

Citation

If you use this model in your research or applications, please cite:

@misc{gemma1b-thai-medical-cot,
  title={Gemma-1B Thai Medical Chain-of-Thought},
  author={Your Name},
  year={2024},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/zoeythanayot/gemma-1b-thai-CoT}}
}

License

This model is licensed under the same terms as the base Gemma model. Please refer to the Gemma License for details.

Contact

For questions or issues regarding this model, please open an issue in the repository or contact the model author.

---

Disclaimer: This model is for research and educational purposes only. Always consult qualified healthcare professionals for medical advice and decisions.