DrafterBench: Benchmarking Large Language Models for Tasks Automation in
  Civil Engineering
Large Language Model (LLM) agents have shown great potential for solving
real-world problems and promise to be a solution for tasks automation in
industry. However, more benchmarks are needed to systematically evaluate
automation agents from an industrial perspective, for example, in Civil
Engineering. Therefore, we propose DrafterBench for the comprehensive
evaluation of LLM agents in the context of technical drawing revision, a
representation task in civil engineering. DrafterBench contains twelve types of
tasks summarized from real-world drawing files, with 46 customized
functions/tools and 1920 tasks in total. DrafterBench is an open-source
benchmark to rigorously test AI agents' proficiency in interpreting intricate
and long-context instructions, leveraging prior knowledge, and adapting to
dynamic instruction quality via implicit policy awareness. The toolkit
comprehensively assesses distinct capabilities in structured data
comprehension, function execution, instruction following, and critical
reasoning. DrafterBench offers detailed analysis of task accuracy and error
statistics, aiming to provide deeper insight into agent capabilities and
identify improvement targets for integrating LLMs in engineering applications.
Our benchmark is available at https://github.com/Eason-Li-AIS/DrafterBench,
with the test set hosted at
https://huggingface.co/datasets/Eason666/DrafterBench.
68785ee1001546c83aa4f967
Yinsheng Li
claimed_verified
2025-07-17T08:22:42.470Z
65c950ebd908bd52a4477116
/avatars/bc6ba0dd2903c7bea37f7b9c40857718.svg
Yinsheng Li
user
Eason666
68785ee1001546c83aa4f968
Zhen Dong
claimed_verified
2025-07-17T08:22:37.102Z
643ba2f725681c3afaa8f05e
https://cdn-avatars.huggingface.co/v1/production/uploads/643ba2f725681c3afaa8f05e/2RnOdmbBM8WHYhiTGG-Cd.jpeg
Zhen Dong
user
zhendongucb
68785ee1001546c83aa4f969
Yi Shao
DrafterBench: Benchmarking Large Language Models for Tasks Automation in
  Civil Engineering
Large Language Model (LLM) agents have shown great potential for solving
real-world problems and promise to be a solution for tasks automation in
industry. However, more benchmarks are needed to systematically evaluate
automation agents from an industrial perspective, for example, in Civil
Engineering. Therefore, we propose DrafterBench for the comprehensive
evaluation of LLM agents in the context of technical drawing revision, a
representation task in civil engineering. DrafterBench contains twelve types of
tasks summarized from real-world drawing files, with 46 customized
functions/tools and 1920 tasks in total. DrafterBench is an open-source
benchmark to rigorously test AI agents' proficiency in interpreting intricate
and long-context instructions, leveraging prior knowledge, and adapting to
dynamic instruction quality via implicit policy awareness. The toolkit
comprehensively assesses distinct capabilities in structured data
comprehension, function execution, instruction following, and critical
reasoning. DrafterBench offers detailed analysis of task accuracy and error
statistics, aiming to provide deeper insight into agent capabilities and
identify improvement targets for integrating LLMs in engineering applications.
Our benchmark is available at https://github.com/Eason-Li-AIS/DrafterBench,
with the test set hosted at
https://huggingface.co/datasets/Eason666/DrafterBench.
DrafterBench is an open-source benchmark for evaluating LLM agents in technical drawing revision, assessing their capabilities in structured data comprehension, function execution, instruction following, and critical reasoning.
Large Language Model (LLM) technical drawing revision structured data comprehension function execution instruction following critical reasoning benchmark open-source implicit policy awareness