transformers safetensors llama text-generation en base_model:TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T base_model:finetune:TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T license:mit autotrain_compatible text-generation-inference endpoints_compatible region:us
---
base_model:
- TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T
language:
- en
library_name: transformers
license: mit
---
# Model Card for ZetaQA-1.1B-XML

## Model Overview
**ZetaQA-1.1B-XML** is a fine-tuned version of **TinyLlama-1.1B**, designed for **structured question answering** with **XML-formatted reasoning**. It specializes in breaking down complex questions into logical steps and providing detailed, structured responses. The model is particularly adept at tasks requiring multi-step reasoning and explainable outputs.

---

## Model Details
- **Architecture**: Transformer-based causal language model
- **Base Model**: TinyLlama-1.1B
- **Fine-Tuning Dataset**: StrategyQA (2,290 examples)
- **Training Framework**: Hugging Face Transformers + TRL
- **Parameter Count**: 1.1 billion
- **License**: Apache 2.0

---

## What This Model Does
The model takes a question as input and generates a **structured response** that includes:
1. **Key Terms**: Identifies important concepts in the question.
2. **Description**: Provides context or definitions for key terms.
3. **Decomposition Steps**: Breaks the question into logical sub-steps.
4. **Relevant Facts**: Lists supporting evidence or facts.
5. **Final Verdict**: Answers the question with "Yes" or "No."
6. **Reasoning**: Explains the final answer based on the decomposition and facts.

### Example:
**Question:** Can penguins fly?

**Response:**
- **Key Term:** Penguins  
- **Description:** Flightless birds native to the southern hemisphere.  
- **Decomposition Steps:**  
  1. Penguins are flightless birds.  
  2. Their wings evolved into flippers.  
- **Relevant Facts:**  
  - All penguin species are flightless.  
  - They use wings for swimming.  
- **Final Verdict:** No  
- **Reasoning:** Penguins' wings have evolved into flippers for swimming, making them physically incapable of flight.

---

## Training Process
1. **Dataset**:
   - **StrategyQA**: A dataset of complex questions requiring multi-step reasoning.
   - Fine-tuned on 2,290 examples with XML-formatted responses.

2. **Preprocessing**:
   - Questions and answers were formatted into XML structures.
   - Special tokens (`<think>`, `</think>`, `<answer>`, `</answer>`) were added to the tokenizer.

3. **Fine-Tuning**:
   - **Framework**: Hugging Face Transformers + TRL (Transformer Reinforcement Learning).
   - **Hardware**: 1x NVIDIA T4 GPU (Google Colab).
   - **Batch Size**: 2 (with gradient accumulation steps of 4).
   - **Learning Rate**: 3e-5.
   - **Epochs**: 3.
   - **Sequence Length**: 1024 tokens.

4. **Evaluation**:
   - Evaluated on a held-out validation set from StrategyQA.
   - Focused on response quality, reasoning accuracy, and XML structure adherence.

---

## Intended Use
- **Primary Use**: Answering complex questions with structured, explainable reasoning.
- **Target Audience**:
  - Researchers studying explainable AI.
  - Developers building educational or decision-support tools.
  - Enterprises needing transparent AI systems.
- **Applications**:
  - Educational platforms (e.g., tutoring systems).
  - Decision support systems (e.g., medical or legal reasoning).
  - Explainable AI pipelines.

---

## Performance
- **Strengths**:
  - Handles multi-step reasoning effectively.
  - Produces human-readable, structured outputs.
  - Lightweight (1.1B parameters) for efficient deployment.
- **Limitations**:
  - May struggle with highly domain-specific questions.
  - Limited by the 1.1B parameter size for extremely complex reasoning.

---

## How to Use
```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained("Kuldeep08/ZetaQA-1.1B-XML")
tokenizer = AutoTokenizer.from_pretrained("Kuldeep08/ZetaQA-1.1B-XML")

# Generate response
question = "Are strawberries a fruit?"
inputs = tokenizer(question, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=200)

# Decode and print response
response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)
```
## Ethical Considerations
Bias: May inherit biases from the base model and training data.

Transparency: XML outputs improve explainability but should be validated for accuracy.

Deployment: Suitable for non-critical applications where errors can be tolerated.