title: LettuceDetect: A Hallucination Detection Framework for RAG Applications
summary: Retrieval Augmented Generation (RAG) systems remain vulnerable to
  hallucinated answers despite incorporating external knowledge sources. We
  present LettuceDetect a framework that addresses two critical limitations in
  existing hallucination detection methods: (1) the context window constraints of
  traditional encoder-based methods, and (2) the computational inefficiency of
  LLM based approaches. Building on ModernBERT's extended context capabilities
  (up to 8k tokens) and trained on the RAGTruth benchmark dataset, our approach
  outperforms all previous encoder-based models and most prompt-based models,
  while being approximately 30 times smaller than the best models. LettuceDetect
  is a token-classification model that processes context-question-answer triples,
  allowing for the identification of unsupported claims at the token level.
  Evaluations on the RAGTruth corpus demonstrate an F1 score of 79.22% for
  example-level detection, which is a 14.8% improvement over Luna, the previous
  state-of-the-art encoder-based architecture. Additionally, the system can
  process 30 to 60 examples per second on a single GPU, making it more practical
  for real-world RAG applications.
paper_authors._id: 67c0536530abbab5c723f2e0
paper_authors.name: Ádám Kovács
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-02T20:18:13.294Z
paper_authors._id: 646264832538819c729e32ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646264832538819c729e32ba/syc-UpPQyR3Nbf-gYndc4.jpeg
paper_authors.fullname: Adam Kovacs
paper_authors.type: user
paper_authors.user: adaamko
paper_authors._id: 67c0536530abbab5c723f2e1
paper_authors.name: Gábor Recski
paper_title: LettuceDetect: A Hallucination Detection Framework for RAG Applications
paper_summary: Retrieval Augmented Generation (RAG) systems remain vulnerable to
  hallucinated answers despite incorporating external knowledge sources. We
  present LettuceDetect a framework that addresses two critical limitations in
  existing hallucination detection methods: (1) the context window constraints of
  traditional encoder-based methods, and (2) the computational inefficiency of
  LLM based approaches. Building on ModernBERT's extended context capabilities
  (up to 8k tokens) and trained on the RAGTruth benchmark dataset, our approach
  outperforms all previous encoder-based models and most prompt-based models,
  while being approximately 30 times smaller than the best models. LettuceDetect
  is a token-classification model that processes context-question-answer triples,
  allowing for the identification of unsupported claims at the token level.
  Evaluations on the RAGTruth corpus demonstrate an F1 score of 79.22% for
  example-level detection, which is a 14.8% improvement over Luna, the previous
  state-of-the-art encoder-based architecture. Additionally, the system can
  process 30 to 60 examples per second on a single GPU, making it more practical
  for real-world RAG applications.
paper_ai_keywords: Retrieval Augmented Generation (RAG) hallucination detection context window constraints encoder-based methods computational inefficiency LLM based approaches ModernBERT extended context capabilities RAGTruth benchmark dataset token-classification model context-question-answer triples unsupported claims example-level detection Luna state-of-the-art encoder-based architecture