AutoDetect: Towards a Unified Framework for Automated Weakness Detection
  in Large Language Models
Although Large Language Models (LLMs) are becoming increasingly powerful,
they still exhibit significant but subtle weaknesses, such as mistakes in
instruction-following or coding tasks. As these unexpected errors could lead to
severe consequences in practical deployments, it is crucial to investigate the
limitations within LLMs systematically. Traditional benchmarking approaches
cannot thoroughly pinpoint specific model deficiencies, while manual
inspections are costly and not scalable. In this paper, we introduce a unified
framework, AutoDetect, to automatically expose weaknesses in LLMs across
various tasks. Inspired by the educational assessment process that measures
students' learning outcomes, AutoDetect consists of three LLM-powered agents:
Examiner, Questioner, and Assessor. The collaboration among these three agents
is designed to realize comprehensive and in-depth weakness identification. Our
framework demonstrates significant success in uncovering flaws, with an
identification success rate exceeding 30% in prominent models such as ChatGPT
and Claude. More importantly, these identified weaknesses can guide specific
model improvements, proving more effective than untargeted data augmentation
methods like Self-Instruct. Our approach has led to substantial enhancements in
popular LLMs, including the Llama series and Mistral-7b, boosting their
performance by over 10% across several benchmarks. Code and data are publicly
available at https://github.com/thu-coai/AutoDetect.
667a39e2dfe717c193c5b3e0
Jiale Cheng
claimed_verified
2024-06-25T07:33:14.266Z
627626d42d26ac639e56f565
/avatars/805c5f909f52656345b8bde486c9fa8f.svg
Jiale Cheng
user
CCCCCC
667a39e2dfe717c193c5b3e1
Yida Lu
admin_assigned
2024-06-25T08:58:44.724Z
6445e2e83a0fa0e98cd0c995
/avatars/5ab8e5fbfaa9e5b456fb94eaa7e62832.svg
Yida Lu
user
yida-lu
667a39e2dfe717c193c5b3e2
Xiaotao Gu
claimed_verified
2025-07-02T09:25:17.074Z
6438bca4a5e10f6d58694b47
/avatars/3aeb25fbc73c5cab1265e13d11adfb76.svg
Xiaotao Gu
user
xgeric
667a39e2dfe717c193c5b3e3
Pei Ke
667a39e2dfe717c193c5b3e4
Xiao Liu
claimed_verified
2024-11-05T16:09:44.560Z
62f098602ca4d32a7cd87aba
https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
Xiao Liu
user
ShawLiu
667a39e2dfe717c193c5b3e5
Yuxiao Dong
admin_assigned
2024-06-25T11:20:27.284Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
667a39e2dfe717c193c5b3e6
Hongning Wang
admin_assigned
2024-06-25T09:00:02.719Z
62ec23fcbd19e355478fc584
/avatars/d7c567ef5f20bb3b9905cb5015d11e12.svg
Hongning Wang
user
howang
667a39e2dfe717c193c5b3e7
Jie Tang
admin_assigned
2024-06-25T11:20:12.918Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
667a39e2dfe717c193c5b3e8
Minlie Huang
AutoDetect: Towards a Unified Framework for Automated Weakness Detection
  in Large Language Models
Although Large Language Models (LLMs) are becoming increasingly powerful,
they still exhibit significant but subtle weaknesses, such as mistakes in
instruction-following or coding tasks. As these unexpected errors could lead to
severe consequences in practical deployments, it is crucial to investigate the
limitations within LLMs systematically. Traditional benchmarking approaches
cannot thoroughly pinpoint specific model deficiencies, while manual
inspections are costly and not scalable. In this paper, we introduce a unified
framework, AutoDetect, to automatically expose weaknesses in LLMs across
various tasks. Inspired by the educational assessment process that measures
students' learning outcomes, AutoDetect consists of three LLM-powered agents:
Examiner, Questioner, and Assessor. The collaboration among these three agents
is designed to realize comprehensive and in-depth weakness identification. Our
framework demonstrates significant success in uncovering flaws, with an
identification success rate exceeding 30% in prominent models such as ChatGPT
and Claude. More importantly, these identified weaknesses can guide specific
model improvements, proving more effective than untargeted data augmentation
methods like Self-Instruct. Our approach has led to substantial enhancements in
popular LLMs, including the Llama series and Mistral-7b, boosting their
performance by over 10% across several benchmarks. Code and data are publicly
available at https://github.com/thu-coai/AutoDetect.
A unified framework, AutoDetect, uses LLM-powered agents to systematically identify and address weaknesses in LLMs, improving their performance across benchmarks.
Large Language Models (LLMs) instruction-following coding tasks systematic limitations investigation traditional benchmarking manual inspections Examiner Questioner Assessor specific model deficiencies identification success rate Self-Instruct model improvements Llama series Mistral-7b