title: DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in
    Digital Forensics and Incident Response
summary: Digital Forensics and Incident Response (DFIR) involves analyzing digital
  evidence to support legal investigations. Large Language Models (LLMs) offer
  new opportunities in DFIR tasks such as log analysis and memory forensics, but
  their susceptibility to errors and hallucinations raises concerns in
  high-stakes contexts. Despite growing interest, there is no comprehensive
  benchmark to evaluate LLMs across both theoretical and practical DFIR domains.
  To address this gap, we present DFIR-Metric, a benchmark with three components:
  (1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice
  questions sourced from industry-standard certifications and official
  documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing
  multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500
  disk and memory forensics cases from the NIST Computer Forensics Tool Testing
  Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their
  accuracy and consistency across trials. We also introduce a new metric, the
  Task Understanding Score (TUS), designed to more effectively evaluate models in
  scenarios where they achieve near-zero accuracy. This benchmark offers a
  rigorous, reproducible foundation for advancing AI in digital forensics. All
  scripts, artifacts, and results are available on the project website at
  https://github.com/DFIR-Metric.
paper_authors._id: 6836a33e64f38b5bf439f9be
paper_authors.name: Bilel Cherif
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:56:58.406Z
paper_authors._id: 64d3db80aea0ccb1b4975d95
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Mi0eKzNp6wKFrqketK-DN.png
paper_authors.fullname: Bilel Cherif
paper_authors.type: user
paper_authors.user: Neo111x
paper_authors._id: 6836a33e64f38b5bf439f9bf
paper_authors.name: Tamas Bisztray
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:57:03.816Z
paper_authors._id: 679d3d70663f033e1a728c17
paper_authors.avatarUrl: /avatars/4ee5580b614c48e3cbb8e123b0ae01a2.svg
paper_authors.fullname: Tamas Bisztray
paper_authors.type: user
paper_authors.user: Sh1n1g4m1
paper_authors._id: 6836a33e64f38b5bf439f9c0
paper_authors.name: Richard A. Dubniczky
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:57:10.313Z
paper_authors._id: 66f1a481fa0f7d6823e95149
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/66f1a481fa0f7d6823e95149/2_r8tJZSr95KkezLNDRu7.jpeg
paper_authors.fullname: Richard A. Dubniczky
paper_authors.type: user
paper_authors.user: dubniczky
paper_authors._id: 6836a33e64f38b5bf439f9c1
paper_authors.name: Aaesha Aldahmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:57:17.034Z
paper_authors._id: 672bb53e7b58518d356e2e18
paper_authors.avatarUrl: /avatars/2945ef62c8efcb7da5a6f1f084589448.svg
paper_authors.fullname: aaesha aldahmani
paper_authors.type: user
paper_authors.user: aaeshaaldahmani
paper_authors._id: 6836a33e64f38b5bf439f9c2
paper_authors.name: Saeed Alshehhi
paper_authors._id: 6836a33e64f38b5bf439f9c3
paper_authors.name: Norbert Tihanyi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-28T12:57:26.722Z
paper_authors._id: 65ae586052e1b2aae48e01fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65ae586052e1b2aae48e01fb/Ga6PC31XgXlfzCteqInNp.png
paper_authors.fullname: Norbert Tihanyi
paper_authors.type: user
paper_authors.user: tihanyin
paper_title: DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in
    Digital Forensics and Incident Response
paper_summary: Digital Forensics and Incident Response (DFIR) involves analyzing digital
  evidence to support legal investigations. Large Language Models (LLMs) offer
  new opportunities in DFIR tasks such as log analysis and memory forensics, but
  their susceptibility to errors and hallucinations raises concerns in
  high-stakes contexts. Despite growing interest, there is no comprehensive
  benchmark to evaluate LLMs across both theoretical and practical DFIR domains.
  To address this gap, we present DFIR-Metric, a benchmark with three components:
  (1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice
  questions sourced from industry-standard certifications and official
  documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing
  multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500
  disk and memory forensics cases from the NIST Computer Forensics Tool Testing
  Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their
  accuracy and consistency across trials. We also introduce a new metric, the
  Task Understanding Score (TUS), designed to more effectively evaluate models in
  scenarios where they achieve near-zero accuracy. This benchmark offers a
  rigorous, reproducible foundation for advancing AI in digital forensics. All
  scripts, artifacts, and results are available on the project website at
  https://github.com/DFIR-Metric.
paper_ai_summary: DFIR-Metric evaluates Large Language Models for digital forensics using a comprehensive benchmark with knowledge assessments, realistic forensic challenges, and practical analysis cases, introducing a Task Understanding Score for near-zero accuracy scenarios.
paper_ai_keywords: Large Language Models (LLMs) DFIR-Metric Knowledge Assessment Realistic Forensic Challenges Practical Analysis Task Understanding Score