safetensors region:us
---
{}
---
# Czech Question Answering Model (Small)

## Overview
A lightweight question answering model fine-tuned for Czech language using LoRA on mBART. This model is optimized for efficiency while maintaining reasonable performance on Czech Q&A tasks.

## Model Details
- Base Model: facebook/mbart-large-cc25
- Size: 15MB (adapter weights)
- Training Method: LoRA fine-tuning
- BLEU Score: 3.61
- Languages: Czech input → Czech output
- Max Sequence Length: 48 tokens

## Training Configuration
```python
LoRA Config:
- r=8 (rank)
- lora_alpha=16
- target_modules=["q_proj", "k_proj", "v_proj", "fc1", "fc2"]
- lora_dropout=0.05

Training Args:
- learning_rate=5e-4
- batch_size=8
- gradient_accumulation_steps=2
- num_train_epochs=1
- warmup_steps=100
- evaluation_steps=100
- save_steps=100
```

## Sample Usage
```python
from transformers import MBartForConditionalGeneration, MBartTokenizer
from safetensors.torch import load_file

# Load base model
model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-cc25")
tokenizer = MBartTokenizer.from_pretrained("facebook/mbart-large-cc25")

# Load fine-tuned weights
adapter_weights = "koushikkanch/small-model/adapter_model.safetensors"
state_dict = load_file(adapter_weights)
model.load_state_dict(state_dict, strict=False)

# Set languages
tokenizer.src_lang = "cs_CZ"
tokenizer.tgt_lang = "cs_CZ"

# Example usage
question = "Je tu obsazeno?"
inputs = tokenizer(question, return_tensors="pt")
outputs = model.generate(**inputs, forced_bos_token_id=tokenizer.lang_code_to_id["cs_CZ"])
answer = tokenizer.batch_decode(outputs, skip_special_tokens=True)[0]
```

## Advantages
- Lightweight (15MB adapter only)
- Fast inference
- Minimal memory requirements
- Easy to deploy
- Quick training time

## Limitations
- Lower BLEU score compared to larger model
- May generate simple or repetitive responses
- Best suited for basic Q&A scenarios

## System Requirements
- CPU: Any modern processor
- RAM: 4GB minimum
- GPU: Optional, can run on CPU
- Disk Space: ~20MB

## Repository Structure
```
small-model/
├── adapter_model.safetensors
├── adapter_config.json
├── tokenizer_config.json
├── special_tokens_map.json
└── README.md

