DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in
  Digital Forensics and Incident Response
Digital Forensics and Incident Response (DFIR) involves analyzing digital
evidence to support legal investigations. Large Language Models (LLMs) offer
new opportunities in DFIR tasks such as log analysis and memory forensics, but
their susceptibility to errors and hallucinations raises concerns in
high-stakes contexts. Despite growing interest, there is no comprehensive
benchmark to evaluate LLMs across both theoretical and practical DFIR domains.
To address this gap, we present DFIR-Metric, a benchmark with three components:
(1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice
questions sourced from industry-standard certifications and official
documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing
multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500
disk and memory forensics cases from the NIST Computer Forensics Tool Testing
Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their
accuracy and consistency across trials. We also introduce a new metric, the
Task Understanding Score (TUS), designed to more effectively evaluate models in
scenarios where they achieve near-zero accuracy. This benchmark offers a
rigorous, reproducible foundation for advancing AI in digital forensics. All
scripts, artifacts, and results are available on the project website at
https://github.com/DFIR-Metric.
6836a33e64f38b5bf439f9be
Bilel Cherif
admin_assigned
2025-05-28T12:56:58.406Z
64d3db80aea0ccb1b4975d95
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Mi0eKzNp6wKFrqketK-DN.png
Bilel Cherif
user
Neo111x
6836a33e64f38b5bf439f9bf
Tamas Bisztray
admin_assigned
2025-05-28T12:57:03.816Z
679d3d70663f033e1a728c17
/avatars/4ee5580b614c48e3cbb8e123b0ae01a2.svg
Tamas Bisztray
user
Sh1n1g4m1
6836a33e64f38b5bf439f9c0
Richard A. Dubniczky
admin_assigned
2025-05-28T12:57:10.313Z
66f1a481fa0f7d6823e95149
https://cdn-avatars.huggingface.co/v1/production/uploads/66f1a481fa0f7d6823e95149/2_r8tJZSr95KkezLNDRu7.jpeg
Richard A. Dubniczky
user
dubniczky
6836a33e64f38b5bf439f9c1
Aaesha Aldahmani
admin_assigned
2025-05-28T12:57:17.034Z
672bb53e7b58518d356e2e18
/avatars/2945ef62c8efcb7da5a6f1f084589448.svg
aaesha aldahmani
user
aaeshaaldahmani
6836a33e64f38b5bf439f9c2
Saeed Alshehhi
6836a33e64f38b5bf439f9c3
Norbert Tihanyi
admin_assigned
2025-05-28T12:57:26.722Z
65ae586052e1b2aae48e01fb
https://cdn-avatars.huggingface.co/v1/production/uploads/65ae586052e1b2aae48e01fb/Ga6PC31XgXlfzCteqInNp.png
Norbert Tihanyi
user
tihanyin
DFIR-Metric: A Benchmark Dataset for Evaluating Large Language Models in
  Digital Forensics and Incident Response
Digital Forensics and Incident Response (DFIR) involves analyzing digital
evidence to support legal investigations. Large Language Models (LLMs) offer
new opportunities in DFIR tasks such as log analysis and memory forensics, but
their susceptibility to errors and hallucinations raises concerns in
high-stakes contexts. Despite growing interest, there is no comprehensive
benchmark to evaluate LLMs across both theoretical and practical DFIR domains.
To address this gap, we present DFIR-Metric, a benchmark with three components:
(1) Knowledge Assessment: a set of 700 expert-reviewed multiple-choice
questions sourced from industry-standard certifications and official
documentation; (2) Realistic Forensic Challenges: 150 CTF-style tasks testing
multi-step reasoning and evidence correlation; and (3) Practical Analysis: 500
disk and memory forensics cases from the NIST Computer Forensics Tool Testing
Program (CFTT). We evaluated 14 LLMs using DFIR-Metric, analyzing both their
accuracy and consistency across trials. We also introduce a new metric, the
Task Understanding Score (TUS), designed to more effectively evaluate models in
scenarios where they achieve near-zero accuracy. This benchmark offers a
rigorous, reproducible foundation for advancing AI in digital forensics. All
scripts, artifacts, and results are available on the project website at
https://github.com/DFIR-Metric.
DFIR-Metric evaluates Large Language Models for digital forensics using a comprehensive benchmark with knowledge assessments, realistic forensic challenges, and practical analysis cases, introducing a Task Understanding Score for near-zero accuracy scenarios.
Large Language Models (LLMs) DFIR-Metric Knowledge Assessment Realistic Forensic Challenges Practical Analysis Task Understanding Score