text-classification en fr ar dataset:OpenLLM-France/Lucie-Training-Dataset dataset:permutans/wdc-common-crawl-embedded-jsonld dataset:calabi-yau-data/ws-5d dataset:mathmadness/MathCoder dataset:AlgorithmicResearchGroup/arxiv_research_code dataset:Vikhrmodels/programming_books_eng dataset:cornstack/cornstack-python-v1 dataset:HuggingFaceH4/MATH-500 dataset:di-zhang-fdu/MATH500 dataset:bezir/MATH-500-multilingual dataset:cais/mmlu dataset:TIGER-Lab/MMLU-Pro dataset:CohereForAI/Global-MMLU dataset:minimario/livecodebench-execute-v2 dataset:akioi/livecodebench_1_to_4 dataset:livecodebench/execution dataset:MatrixStudio/Codeforces-Python-Submissions dataset:DenCT/codeforces-problems-7k dataset:evanellis/Codeforces-LLM-Generations dataset:evanellis/Codeforces-LLM-Generations_with_completions dataset:Maxwell-Jia/AIME_2024 dataset:HuggingFaceH4/aime_2024 region:us
---
datasets:
- OpenLLM-France/Lucie-Training-Dataset
- permutans/wdc-common-crawl-embedded-jsonld
- calabi-yau-data/ws-5d
- mathmadness/MathCoder
- AlgorithmicResearchGroup/arxiv_research_code
- Vikhrmodels/programming_books_eng
- cornstack/cornstack-python-v1
- HuggingFaceH4/MATH-500
- di-zhang-fdu/MATH500
- bezir/MATH-500-multilingual
- cais/mmlu
- TIGER-Lab/MMLU-Pro
- CohereForAI/Global-MMLU
- minimario/livecodebench-execute-v2
- akioi/livecodebench_1_to_4
- livecodebench/execution
- MatrixStudio/Codeforces-Python-Submissions
- DenCT/codeforces-problems-7k
- evanellis/Codeforces-LLM-Generations
- evanellis/Codeforces-LLM-Generations_with_completions
- Maxwell-Jia/AIME_2024
- HuggingFaceH4/aime_2024
language:
- en
- fr
- ar
metrics:
- accuracy
pipeline_tag: text-classification
---
<!---
Copyright 2025 EGen V1. All rights reserved.

Licensed under the EGen License v0.1
-->

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://imgg.io/images/2025/02/23/db70bc9a08fcaaa5765c04dfa3fa7301.png">
    <source media="(prefers-color-scheme: light)" srcset="https://imgg.io/images/2025/02/23/db70bc9a08fcaaa5765c04dfa3fa7301.png">
    <img alt="EGen V1 Transformers" src="https://imgg.io/images/2025/02/23/db70bc9a08fcaaa5765c04dfa3fa7301.png" width="200" height="200">
  </picture>
  <br/>
  <h2 align="center">EGen V1 Implementation</h2>
</p>

<div align="center" style="line-height: 1;">
  <a href="https://huggingface.co/ErebusTN/EGen_V1" target="_blank" style="margin: 2px;">
    <img alt="Hugging Face" src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-EGen%20V1-ffc107?color=ffc107&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://github.com/ErebusTN" target="_blank" style="margin: 2px;">
    <img alt="GitHub" src="https://img.shields.io/badge/GitHub-EGen%20V1-181717?logo=github&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="mailto:mouhebga62@gmail.com" target="_blank" style="margin: 2px;">
    <img alt="Email" src="https://img.shields.io/badge/Email-mouhebga62@gmail.com-brightgreen?logo=gmail&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
  <a href="https://huggingface.co/ErebusTN/EGen_V1/blob/main/LICENSE" style="margin: 2px;">
    <img alt="License" src="https://img.shields.io/badge/License-EGen%20V1-f5de53?&color=f5de53&logoColor=white" style="display: inline-block; vertical-align: middle;"/>
  </a>
</div>

<h3 align="center">
    <p>Enterprise-Grade AI Models with THL-150 Architecture</p>
</h3>

## 📖 Table of Contents
- [Project Setup Guide](#-project-setup-guide)
- [Key Features](#-key-features)
- [Getting Started](#-getting-started)
- [Model Architecture](#-model-architecture)
- [Performance Benchmarks](#-performance-benchmarks)
- [Advanced Usage](#-advanced-usage)
- [Training Guide](#-training-guide)
- [Deployment Options](#-deployment-options)
- [Enterprise Features](#-enterprise-features)
- [Troubleshooting](#-troubleshooting)
- [Contributing](#-contributing)
- [License](#-license)
- [Citation](#-citation)

## 🛠️ Project Setup Guide

### 1. Repository Setup
```bash
# Clone the repository
git clone https://github.com/ErebusTN/transformers.git
cd transformers

# Initialize project structure
mkdir -p my_egen_project/{configs,data,models,scripts}
touch my_egen_project/Dockerfile my_egen_project/requirements.txt

# Create and activate virtual environment
python -m venv .egenenv
source .egenenv/bin/activate  # Linux/Mac
# .\egenenv\Scripts\activate  # Windows
```

### 2. Dependency Installation
```bash
# Install core packages
pip install -e .[cuda,testing]
pip install EGen-V1 torch>=2.1.1 accelerate

# Development tools
pip install black flake8 isort pytest
```

### 3. Docker Configuration
```dockerfile
# Dockerfile
FROM nvidia/cuda:12.2.0-devel-ubuntu22.04

RUN apt-get update && apt-get install -y \
    python3.10 \
    python3-pip \
    git \
    && rm -rf /var/lib/apt/lists/*

RUN git clone https://github.com/ErebusTN/transformers.git
WORKDIR /transformers

RUN pip install -e .[cuda] && \
    pip install EGen-V1 torch==2.1.1 accelerate

CMD ["python3"]
```

### 4. Verification Script
```python
# verification_test.py
from transformers import THL150ForCausalLM, THL150Tokenizer

model = THL150ForCausalLM.from_pretrained(
    'ErebusTN/EGen_V1',
    device_map='auto',
    torch_dtype='auto'
)
tokenizer = THL150Tokenizer.from_pretrained('ErebusTN/EGen_V1')

print("✔ Installation verified successfully!")
```

## ✨ Key Features
- **32k Context Window** with dynamic attention scaling
- **Hybrid Attention Mechanisms**: GQA, MQA, Sliding Window
- **Enhanced RoPE Embeddings** with dynamic scaling
- **Multi-modal Support**: Text, Code, Mathematics
- **Enterprise-Grade Security**: Model watermarking, audit logging

## 🚀 Getting Started

### Basic Usage
```python
from transformers import THL150ForCausalLM, THL150Tokenizer

model = THL150ForCausalLM.from_pretrained(
    'ErebusTN/EGen_V1',
    device_map='auto',
    torch_dtype=torch.bfloat16
)
tokenizer = THL150Tokenizer.from_pretrained('ErebusTN/EGen_V1')

inputs = tokenizer("AI's future is", return_tensors="pt", max_length=32768)
outputs = model.generate(**inputs, max_new_tokens=100)
print(tokenizer.decode(outputs[0]))
```

### Model Heads
```python
# Sequence Classification
from transformers import THL150ForSequenceClassification
model = THL150ForSequenceClassification.from_pretrained('ErebusTN/EGen_V1')

# Question Answering
from transformers import THL150ForQuestionAnswering
model = THL150ForQuestionAnswering.from_pretrained('ErebusTN/EGen_V1')

# Token Classification
from transformers import THL150ForTokenClassification 
model = THL150ForTokenClassification.from_pretrained('ErebusTN/EGen_V1')
```

## 🧠 Model Architecture

### Core Components
```text
src/transformers/models/thl_150/
├── __init__.py
├── configuration_thl_150.py
├── modeling_thl_150.py
├── tokenization_thl_150.py
└── tokenization_thl_150_fast.py
```

### Attention Mechanism
```python
class THL150Attention(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.window_size = config.window_size
        self.num_heads = config.num_attention_heads
        self.head_dim = config.hidden_size // config.num_attention_heads
        
        # QKV Projections
        self.q_proj = nn.Linear(config.hidden_size, config.hidden_size)
        self.k_proj = nn.Linear(config.hidden_size, config.hidden_size)
        self.v_proj = nn.Linear(config.hidden_size, config.hidden_size)
        
        # Rotary Positional Embeddings
        self.rotary_emb = RotaryEmbedding(dim=self.head_dim)
        
    def forward(self, hidden_states, attention_mask=None):
        # Sliding window attention implementation
        ...
```

## 📊 Performance Benchmarks

| Metric              | EGen V1 (32k) | LLaMA-2 13B | Mistral 7B |
|---------------------|---------------|-------------|------------|
| MMLU Accuracy       | 72.3%         | 69.8%       | 70.5%      |
| CodeGen (HumanEval) | 65.1%         | 58.4%       | 61.2%      |
| Inference Latency   | 38ms/token    | 45ms/token  | 42ms/token |
| Training Throughput | 1.2 samples/s | 0.9 samples/s| 1.1 samples/s |

## 🔧 Advanced Usage

### Quantization
```python
from transformers import BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = THL150ForCausalLM.from_pretrained(
    'ErebusTN/EGen_V1',
    quantization_config=quant_config,
    device_map='auto'
)
```

### Custom Configuration
```python
from transformers import THL150Config

custom_config = THL150Config(
    window_size=1024,
    rope_scaling={
        "type": "linear",
        "factor": 2.0,
        "original_max_position_embeddings": 32768
    },
    num_key_value_heads=8,
    hidden_act="silu"
)
```

## 🎓 Training Guide

### Multi-GPU Training
```bash
deepspeed --num_gpus 8 train.py \
  --model_name ErebusTN/EGen_V1 \
  --dataset OpenLLM-France/Lucie-Training-Dataset \
  --per_device_train_batch_size 2 \
  --gradient_accumulation_steps 16 \
  --learning_rate 2e-5 \
  --fp16 \
  --deepspeed configs/ds_config.json
```

## 🚢 Deployment

### ONNX Export
```python
from optimum.onnxruntime import ORTModelForCausalLM

model = ORTModelForCausalLM.from_pretrained(
    "ErebusTN/EGen_V1",
    export=True,
    provider="CUDAExecutionProvider"
)
model.save_pretrained("./onnx_model/")
```

### TensorRT Optimization
```bash
trtexec --onnx=model.onnx \
        --saveEngine=model.plan \
        --fp16 \
        --memPoolSize=workspace:4096 \
        --builderOptimizationLevel=5
```

## 🏢 Enterprise Features

| Feature              | Description                           |
|----------------------|---------------------------------------|
| Model Watermarking   | Cryptographic model fingerprinting   |
| Audit Logging        | GDPR-compliant activity tracking     |
| Dynamic Batching     | Automatic batch size optimization    |
| Secure API Endpoints | TLS 1.3 + OAuth2 authentication      |

## 🛠️ Troubleshooting

### Common Issues

**CUDA Setup:**
```bash
# Verify CUDA
nvidia-smi
python -c "import torch; print(torch.cuda.is_available())"

# Reinstall with specific CUDA version
pip install torch==2.1.1+cu121 --extra-index-url https://download.pytorch.org/whl/cu121
```

**Memory Optimization:**
```python
model = THL150ForCausalLM.from_pretrained(
    'ErebusTN/EGen_V1',
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)
```

## 🤝 Contributing

### Workflow
```bash
# Create feature branch
git checkout -b feature/new-optimization

# Run tests
pytest tests/models/thl_150/ --cov=transformers.models.thl_150

# Submit PR
git push origin feature/new-optimization
```
<center>

## 📜 License
[EGen License v0.1](https://huggingface.co/ErebusTN/EGen_V1/blob/main/LICENSE.md) |
 [Contact for commercial use](mailto:mouhebga62@gmail.com)
</center>

## 📖 Citation

```bibtex
@inproceedings{egen2024,
    title = {EGen V1: Enterprise Language Model with THL-150 Architecture},
    author = {Your Name and Team},
    year = {2024},
    publisher = {Your Organization},
    url = {https://huggingface.co/ErebusTN/EGen_V1}
}
```

---
<div align="center">
  <h3>🌟 Community Resources</h3>
  <p>
    <a href="https://github.com/ErebusTN/transformers/issues">GitHub Issues</a> •
    <a href="https://huggingface.co/ErebusTN/EGen_V1/discussions">HF Discussions</a> •
    <a href="#">Discord Server (soon)</a>
  </p>

  <h3>📅 Development Roadmap</h3>
  <ul align="left">
    <li>[Q3 2024] Expand to 100k context window</li>
    <li>[Q4 2024] Add multimodal capabilities</li>
    <li>[Q1 2025] Optimize for specialized hardware</li>
  </ul>
</div>

<div align="center">
  <h3>EGen V1 - Redefining AI Boundaries</h3>
  <p>
    <a href="https://huggingface.co/ErebusTN/EGen_V1/blob/main/Optimization-Guide.md">Performance Tuning</a> •
    <a href="https://huggingface.co/ErebusTN/EGen_V1/blob/main/Guidelines.md">Contribution Guide</a> •
    <a href="https://huggingface.co/ErebusTN/EGen_V1/blob/main/ARCHITECTURE.md">Architecture Details</a>
  </p>
</div>
