|
|
--- |
|
|
license: mit |
|
|
language: |
|
|
- de |
|
|
base_model: |
|
|
- EuroBERT/EuroBERT-610m |
|
|
pipeline_tag: token-classification |
|
|
tags: |
|
|
- token classification |
|
|
- hallucination detection |
|
|
- transformers |
|
|
- question answer |
|
|
datasets: |
|
|
- KRLabsOrg/ragtruth-de-translated |
|
|
--- |
|
|
|
|
|
# LettuceDetect: German Hallucination Detection Model |
|
|
|
|
|
<p align="center"> |
|
|
<img src="https://github.com/KRLabsOrg/LettuceDetect/blob/feature/cn_llm_eval/assets/lettuce_detective_multi.png?raw=true" alt="LettuceDetect Logo" width="400"/> |
|
|
</p> |
|
|
|
|
|
**Model Name:** KRLabsOrg/lettucedect-610m-eurobert-de-v1 |
|
|
**Organization:** KRLabsOrg |
|
|
**Github:** https://github.com/KRLabsOrg/LettuceDetect |
|
|
|
|
|
## Overview |
|
|
|
|
|
LettuceDetect is a transformer-based model for hallucination detection on context and answer pairs, designed for multilingual Retrieval-Augmented Generation (RAG) applications. This model is built on **EuroBERT-610M**, which has been specifically chosen for its extended context support (up to **8192 tokens**) and strong multilingual capabilities. This long-context capability is critical for tasks where detailed and extensive documents need to be processed to accurately determine if an answer is supported by the provided context. |
|
|
|
|
|
**This is our German large model utilizing EuroBERT-610M architecture** |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Architecture:** EuroBERT-610M with extended context support (up to 8192 tokens) |
|
|
- **Task:** Token Classification / Hallucination Detection |
|
|
- **Training Dataset:** RagTruth-DE (translated from the original RAGTruth dataset) |
|
|
- **Language:** German |
|
|
|
|
|
## How It Works |
|
|
|
|
|
The model is trained to identify tokens in the German answer text that are not supported by the given context. During inference, the model returns token-level predictions which are then aggregated into spans. This allows users to see exactly which parts of the answer are considered hallucinated. |
|
|
|
|
|
## Usage |
|
|
|
|
|
### Installation |
|
|
|
|
|
Install the 'lettucedetect' repository |
|
|
|
|
|
```bash |
|
|
pip install lettucedetect |
|
|
``` |
|
|
|
|
|
### Using the model |
|
|
|
|
|
```python |
|
|
from lettucedetect.models.inference import HallucinationDetector |
|
|
|
|
|
# For a transformer-based approach: |
|
|
detector = HallucinationDetector( |
|
|
method="transformer", |
|
|
model_path="KRLabsOrg/lettucedect-610m-eurobert-de-v1", |
|
|
lang="de", |
|
|
trust_remote_code=True |
|
|
) |
|
|
|
|
|
contexts = ["Frankreich ist ein Land in Europa. Die Hauptstadt von Frankreich ist Paris. Die Bevölkerung Frankreichs beträgt 67 Millionen."] |
|
|
question = "Was ist die Hauptstadt von Frankreich? Wie groß ist die Bevölkerung Frankreichs?" |
|
|
answer = "Die Hauptstadt von Frankreich ist Paris. Die Bevölkerung Frankreichs beträgt 69 Millionen." |
|
|
|
|
|
# Get span-level predictions indicating which parts of the answer are considered hallucinated. |
|
|
predictions = detector.predict(context=contexts, question=question, answer=answer, output_format="spans") |
|
|
print("Predictions:", predictions) |
|
|
|
|
|
# Predictions: [{'start': 41, 'end': 88, 'confidence': 0.9873546123504639, 'text': ' Die Bevölkerung Frankreichs beträgt 69 Millionen.'}] |
|
|
``` |
|
|
|
|
|
## Performance |
|
|
|
|
|
**Results on Translated RAGTruth-DE** |
|
|
|
|
|
We evaluate our German models on translated versions of the [RAGTruth](https://aclanthology.org/2024.acl-long.585/) dataset. The EuroBERT-610M German model achieves an F1 score of 74.95%, significantly outperforming prompt-based methods like GPT-4.1-mini (60.91%) with a substantial improvement of +14.04 percentage points. |
|
|
|
|
|
For detailed performance metrics across different languages, see the table below: |
|
|
|
|
|
| Language | Model | Precision (%) | Recall (%) | F1 (%) | GPT-4.1-mini F1 (%) | Δ F1 (%) | |
|
|
|----------|-----------------|---------------|------------|--------|---------------------|----------| |
|
|
| German | EuroBERT-210M | 66.70 | 66.70 | 66.70 | 60.91 | +5.79 | |
|
|
| German | EuroBERT-610M | **77.04** | **72.96** | **74.95** | 60.91 | **+14.04** | |
|
|
|
|
|
While the 610M variant requires more computational resources, it delivers substantially higher performance with over 8 percentage points improvement in F1 score compared to the 210M model. |
|
|
|
|
|
### Manual Validation |
|
|
|
|
|
We performed additional validation on a manually reviewed set of 300 examples covering all task types from the data (QA, summarization, data-to-text). The EuroBERT-610M German model showed strong performance with an F1 score of 71.79% on this curated dataset. |
|
|
|
|
|
| Model | Precision (%) | Recall (%) | F1 (%) | |
|
|
|---------------|---------------|------------|--------| |
|
|
| EuroBERT-210M | 68.32 | 68.32 | 68.32 | |
|
|
| EuroBERT-610M | **74.47** | 69.31 | **71.79** | |
|
|
|
|
|
## Citing |
|
|
|
|
|
If you use the model or the tool, please cite the following paper: |
|
|
|
|
|
```bibtex |
|
|
@misc{Kovacs:2025, |
|
|
title={LettuceDetect: A Hallucination Detection Framework for RAG Applications}, |
|
|
author={Ádám Kovács and Gábor Recski}, |
|
|
year={2025}, |
|
|
eprint={2502.17125}, |
|
|
archivePrefix={arXiv}, |
|
|
primaryClass={cs.CL}, |
|
|
url={https://arxiv.org/abs/2502.17125}, |
|
|
} |
|
|
``` |
