|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- en |
|
|
pipeline_tag: text-generation |
|
|
library_name: transformers |
|
|
--- |
|
|
|
|
|
# 🧠 Model Card: Sam-2.5-4 (Pro Version) |
|
|
|
|
|
## 📌 Overview |
|
|
|
|
|
**Sam-2.5-4** is the Pro continuation of the Sam-2.5 architecture series, designed for modular, multi-domain reasoning across math, dialogue, code, and open-domain tasks. It builds directly on **Sam-2.5-3**, continuing training for four additional epochs to deepen convergence, reduce domain bias, and improve generalization. |
|
|
|
|
|
This model is optimized for transparency, ablation-readiness, and deployment on both high-resource and low-resource devices (including Raspberry Pi). |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧬 Model Lineage |
|
|
|
|
|
| Version | Description | |
|
|
|---------------|------------------------------------------------------------------------------| |
|
|
| **Sam-2.5-2** | GSM8K-heavy fine-tune; overfit to math; lacked domain balance | |
|
|
| **Sam-2.5-3** | Emergency patch; retrained from scratch on 4 datasets; balanced capabilities | |
|
|
| **Sam-2.5-4** | Pro version; continued training for 4 epochs; refined convergence and fluency| |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧠 Architecture |
|
|
|
|
|
- Transformer-based, modular design |
|
|
- Registry-driven domain tagging and ablation toggles |
|
|
- Shape-adaptive loss functions with domain-aware diagnostics |
|
|
- Quantization-ready for Pi deployment |
|
|
- Verbose logging for batch-level feedback and anomaly tracing |
|
|
- Memory-safe serialization via `safetensors` |
|
|
|
|
|
--- |
|
|
|
|
|
## 📚 Training Datasets |
|
|
|
|
|
| Dataset | Domain Focus | |
|
|
|------------------------|----------------------------------| |
|
|
| **GSM8K** | Mathematical reasoning | |
|
|
| **MultiWOZ** | Multi-turn dialogue & task flow | |
|
|
| **Alpaca-Code-Cleaned**| Code generation & logic | |
|
|
| **UltraChat-200k** | Open-domain conversation | |
|
|
|
|
|
- Datasets were concatenated, shuffled, and tagged for domain awareness |
|
|
- Replay and mixing strategies used to balance underrepresented domains |
|
|
- Training spanned **9 total epochs** (5 in -3, 4 in -4) |
|
|
|
|
|
--- |
|
|
|
|
|
## 📈 Performance Summary |
|
|
|
|
|
| Metric | Value (Epoch 8–9) | |
|
|
|-------------------------|----------------------------------| |
|
|
| **Validation Loss** | ↓ 2.95 (avg across domains) | |
|
|
| **Max Domain Loss** | < 3.4 (no domain exceeded) | |
|
|
| **Math Bias** | Resolved (loss spikes absorbed) | |
|
|
| **Dialogue Coherence** | Improved (MultiWOZ eval) | |
|
|
| **Code Determinism** | Increased (Alpaca eval) | |
|
|
| **Open-Domain Fluency** | Fewer hallucinations, better grounding | |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧪 Evaluation & Diagnostics |
|
|
|
|
|
- Loss spikes in early epochs traced to GSM8K; resolved by epoch 6 |
|
|
- Batch-level diagnostics printed per domain and token type |
|
|
- Attention stability improved on long-context prompts |
|
|
- Token transitions cleaner across dialogue and code tasks |
|
|
- Validation curve shows smooth convergence post-epoch 5 |
|
|
|
|
|
--- |
|
|
|
|
|
## 🧩 Deployment Notes |
|
|
|
|
|
- Compatible with Raspberry Pi (quantized + safetensors) |
|
|
- Supports CLI-based training diagnostics (loss, ETA, memory) |
|
|
- Registry hooks enable domain-specific ablation and extension |
|
|
- Ideal for benchmarking on GSM8K, MultiWOZ, UltraChat, and custom blends |
|
|
|
|
|
--- |
|
|
|
|
|
## 🤝 Intended Use |
|
|
|
|
|
- Research on modular Transformer architectures |
|
|
- Benchmarking across reasoning, dialogue, and code domains |
|
|
- Deployment on constrained hardware (e.g. Pi, ARM) |
|
|
- Community-driven extension and ablation testing |
|
|
|
|
|
--- |
|
|
|
|
|
## ⚠️ Limitations |
|
|
|
|
|
- Still sensitive to prompt phrasing in edge cases |
|
|
- Long-context performance may degrade beyond 2k tokens |
|
|
- Requires domain tags for optimal generalization |
|
|
- Not trained on multimodal inputs (text-only) |
|
|
|
|
|
--- |
|
|
|
|
|
## 🙌 Acknowledgments |
|
|
|
|
|
Thanks to the open-source community, dataset curators, and contributors who helped shape Sam-2.5-4. This release reflects our shared commitment to transparent, inspectable, and extensible AI. |
