KhasiBERT: Foundational Language Model for Khasi
KhasiBERT is the first foundational language model for the Khasi language, trained on 3.6 million sentences using the RoBERTa architecture. This model serves as the foundation for all downstream Khasi NLP tasks including text classification, sentiment analysis, question answering, and language generation.
Model Overview
| Attribute | Value |
|---|---|
| Model Name | KhasiBERT |
| Version | 1.0.0 |
| Architecture | RoBERTa-base |
| Parameters | 110,652,416 |
| Model Size | 421 MB |
| Language | Khasi (kha) |
| Language Family | Austroasiatic |
| Training Data | 3,621,116 sentences |
| Vocabulary Size | 32,000 tokens |
| Max Sequence Length | 512 tokens |
| Training Time | ~4 hours |
| GPU Used | NVIDIA RTX A6000 (48GB) |
Model Architecture
KhasiBERT follows the RoBERTa-base architecture with the following specifications:
| Component | Configuration |
|---|---|
| Transformer Layers | 12 |
| Hidden Size | 768 |
| Attention Heads | 12 |
| Intermediate Size | 3,072 |
| Activation Function | GELU |
| Dropout | 0.1 |
| Layer Norm Epsilon | 1e-12 |
| Max Position Embeddings | 514 |
Training Methodology
Training Objective
KhasiBERT was trained using Masked Language Modeling (MLM), where 15% of input tokens are randomly masked and the model learns to predict these masked tokens based on bidirectional context.
Training Configuration
| Hyperparameter | Value |
|---|---|
| Training Objective | Masked Language Modeling |
| Masking Probability | 15% |
| Optimizer | AdamW |
| Learning Rate | 5e-5 |
| Learning Rate Schedule | Linear with warmup |
| Warmup Steps | 5,000 |
| Weight Decay | 0.01 |
| Batch Size | 24 |
| Gradient Accumulation | 1 |
| Training Epochs | 1 |
| Total Training Steps | 150,880 |
| Mixed Precision | FP16 |
| Hardware | NVIDIA RTX A6000 (48GB) |
Tokenization
A custom Byte-Level BPE tokenizer was trained specifically on the Khasi corpus with:
- Vocabulary size: 32,000 tokens
- Special tokens:
<s>,</s>,<pad>,<unk>,<mask> - Trained on the complete Khasi dataset for optimal language coverage
Training Data
Dataset Statistics
| Statistic | Value |
|---|---|
| Total Sentences | 3,621,116 |
| Average Sentence Length | 83 characters |
| Estimated Total Tokens | ~50-70 million |
| Data Quality | High-quality, deduplicated |
| Language Coverage | Comprehensive Khasi text |
Data Preprocessing
- Exact duplicate removal
- Near-duplicate removal (80% similarity threshold)
- Length filtering (10-500 characters)
- Text normalization and cleaning
- Quality validation
Usage
Quick Start
from transformers import RobertaForMaskedLM, RobertaTokenizerFast, pipeline
# Load model and tokenizer
model = RobertaForMaskedLM.from_pretrained('MWirelabs/khasibert')
tokenizer = RobertaTokenizerFast.from_pretrained('MWirelabs/khasibert')
# Create fill-mask pipeline
fill_mask = pipeline('fill-mask', model=model, tokenizer=tokenizer)
# Example usage
text = 'Ka Meghalaya ka <mask> ha ka jingpyrkhat jong ki Khasi.'
results = fill_mask(text)
print(results)
Fine-tuning for Downstream Tasks
from transformers import RobertaForSequenceClassification
# For text classification
model = RobertaForSequenceClassification.from_pretrained(
'MWirelabs/khasibert',
num_labels=2 # Adjust for your task
)
# Fine-tune for sentiment analysis, document classification, etc.
Model Performance
Masked Language Modeling Results
KhasiBERT demonstrates strong contextual understanding in Khasi:
| Test Case | Input | Top Prediction | Confidence |
|---|---|---|---|
| Question Context | Phi lah bam | bha | 6.8% |
| Location Context | Ka shnong jongngi ka don ha pdeng ki | khlaw | 7.1% |
| Place Reference | Ngan sa leit kai sha | Delhi | 10.6% |
| Action Context | Ngi donkam ban leit sha iew ban thied | jingthied | 25.3% |
| Gratitude Expression | Khublei shibun na ka bynta ka jingiarap jong | phi | 44.7% |
Key Strengths
- Authentic Khasi understanding with contextually appropriate predictions
- Pronoun recognition - correctly predicts "phi" (you) in gratitude expressions (44.7%)
- Semantic relationships - predicts "jingthied" related to "thied" (25.3%)
- Place name recognition - identifies proper nouns like "Delhi" (10.6%)
- Grammatical structure awareness across question, location, and action contexts
Applications
KhasiBERT serves as the foundation for various Khasi NLP applications:
Supported Tasks
- Text Classification: Document categorization, topic modeling
- Sentiment Analysis: Opinion mining in Khasi text
- Named Entity Recognition: Person, place, organization extraction
- Question Answering: Khasi reading comprehension systems
- Text Generation: Coherent Khasi text creation
- Language Understanding: Chatbots and virtual assistants
- Machine Translation: English-Khasi translation systems
Use Cases
- Educational technology for Khasi language learning
- Government services in Meghalaya
- Cultural preservation and digital humanities
- Social media monitoring and analysis
- Content recommendation systems
Regional Significance
For Meghalaya State
- Digital Government Services: Enables Khasi language interfaces for e-governance
- Educational Technology: Powers AI-driven Khasi language learning platforms
- Cultural Preservation: Digitally preserves and promotes Khasi linguistic heritage
- Economic Development: Creates foundation for local language tech industry
For Northeast India
- Linguistic Diversity: Supports AI development for Northeast India's rich language landscape
- Digital Inclusion: Ensures indigenous communities aren't left behind in AI revolution
- Research Hub: Positions Meghalaya as center for indigenous language AI research
Technical Requirements
Hardware Requirements
| Task | RAM | GPU Memory | GPU |
|---|---|---|---|
| Inference (CPU) | 4GB | - | - |
| Inference (GPU) | 8GB | 2GB | Any CUDA GPU |
| Fine-tuning | 16GB | 8GB | RTX 3080+ |
| Full Training | 32GB | 24GB+ | RTX 4090/A6000+ |
Software Requirements
- Python 3.8+
- PyTorch 1.9+
- Transformers 4.20+
- CUDA 11.0+ (for GPU)
Research Context
Significance
KhasiBERT represents a significant advancement in low-resource NLP:
- First foundational model for the Khasi language
- Enables NLP research for 1.4+ million Khasi speakers
- Preserves linguistic heritage through AI technology
- Demonstrates efficient training methodology for resource-constrained scenarios
Limitations
- Trained on 1 epoch (future versions may benefit from additional training)
- Performance may vary on highly domain-specific text
- Requires task-specific fine-tuning for optimal performance
- May not capture all dialectal variations of Khasi
Citation
If you use KhasiBERT in your research or applications, please cite:
@misc{khasibert2025,
title = {KhasiBERT v1.0: A Foundational Language Model for Khasi},
author = {MWire Labs},
year = {2025},
publisher = {Zenodo},
doi = {10.5281/zenodo.17063992},
url = {https://doi.org/10.5281/zenodo.17063992}
}
Contact & Support
- Organization: MWirelabs
- Model Repository: https://huggingface.co/MWirelabs/khasibert
- Issues: Please report issues through the Hugging Face model page
Acknowledgments
- Training conducted on NVIDIA RTX A6000 GPU
- Built using the Transformers library by Hugging Face
- Inspired by the success of foundational models for major languages
- Dedicated to the preservation and advancement of the Khasi language
License
This model is released under the Creative Commons BY-NC 4.0 License. You are free to:
- Use for non-commercial research and education
- Modify and distribute for non-commercial purposes
- Create derivative works for research
Commercial Use: Contact MWirelabs for commercial licensing agreements. Attribution Required: Please provide appropriate credit to MWirelabs when using this model.
KhasiBERT: Bridging traditional Khasi language with modern AI technology.
- Downloads last month
- 19