title: AutoDetect: Towards a Unified Framework for Automated Weakness Detection
    in Large Language Models
summary: Although Large Language Models (LLMs) are becoming increasingly powerful,
  they still exhibit significant but subtle weaknesses, such as mistakes in
  instruction-following or coding tasks. As these unexpected errors could lead to
  severe consequences in practical deployments, it is crucial to investigate the
  limitations within LLMs systematically. Traditional benchmarking approaches
  cannot thoroughly pinpoint specific model deficiencies, while manual
  inspections are costly and not scalable. In this paper, we introduce a unified
  framework, AutoDetect, to automatically expose weaknesses in LLMs across
  various tasks. Inspired by the educational assessment process that measures
  students' learning outcomes, AutoDetect consists of three LLM-powered agents:
  Examiner, Questioner, and Assessor. The collaboration among these three agents
  is designed to realize comprehensive and in-depth weakness identification. Our
  framework demonstrates significant success in uncovering flaws, with an
  identification success rate exceeding 30% in prominent models such as ChatGPT
  and Claude. More importantly, these identified weaknesses can guide specific
  model improvements, proving more effective than untargeted data augmentation
  methods like Self-Instruct. Our approach has led to substantial enhancements in
  popular LLMs, including the Llama series and Mistral-7b, boosting their
  performance by over 10% across several benchmarks. Code and data are publicly
  available at https://github.com/thu-coai/AutoDetect.
paper_authors._id: 667a39e2dfe717c193c5b3e0
paper_authors.name: Jiale Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T07:33:14.266Z
paper_authors._id: 627626d42d26ac639e56f565
paper_authors.avatarUrl: /avatars/805c5f909f52656345b8bde486c9fa8f.svg
paper_authors.fullname: Jiale Cheng
paper_authors.type: user
paper_authors.user: CCCCCC
paper_authors._id: 667a39e2dfe717c193c5b3e1
paper_authors.name: Yida Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:58:44.724Z
paper_authors._id: 6445e2e83a0fa0e98cd0c995
paper_authors.avatarUrl: /avatars/5ab8e5fbfaa9e5b456fb94eaa7e62832.svg
paper_authors.fullname: Yida Lu
paper_authors.type: user
paper_authors.user: yida-lu
paper_authors._id: 667a39e2dfe717c193c5b3e2
paper_authors.name: Xiaotao Gu
paper_authors._id: 667a39e2dfe717c193c5b3e3
paper_authors.name: Pei Ke
paper_authors._id: 667a39e2dfe717c193c5b3e4
paper_authors.name: Xiao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T16:09:44.560Z
paper_authors._id: 62f098602ca4d32a7cd87aba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1659934844914-62f098602ca4d32a7cd87aba.jpeg
paper_authors.fullname: Xiao Liu
paper_authors.type: user
paper_authors.user: ShawLiu
paper_authors._id: 667a39e2dfe717c193c5b3e5
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T11:20:27.284Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 667a39e2dfe717c193c5b3e6
paper_authors.name: Hongning Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T09:00:02.719Z
paper_authors._id: 62ec23fcbd19e355478fc584
paper_authors.avatarUrl: /avatars/d7c567ef5f20bb3b9905cb5015d11e12.svg
paper_authors.fullname: Hongning Wang
paper_authors.type: user
paper_authors.user: howang
paper_authors._id: 667a39e2dfe717c193c5b3e7
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T11:20:12.918Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_authors._id: 667a39e2dfe717c193c5b3e8
paper_authors.name: Minlie Huang
paper_title: AutoDetect: Towards a Unified Framework for Automated Weakness Detection
    in Large Language Models
paper_summary: Although Large Language Models (LLMs) are becoming increasingly powerful,
  they still exhibit significant but subtle weaknesses, such as mistakes in
  instruction-following or coding tasks. As these unexpected errors could lead to
  severe consequences in practical deployments, it is crucial to investigate the
  limitations within LLMs systematically. Traditional benchmarking approaches
  cannot thoroughly pinpoint specific model deficiencies, while manual
  inspections are costly and not scalable. In this paper, we introduce a unified
  framework, AutoDetect, to automatically expose weaknesses in LLMs across
  various tasks. Inspired by the educational assessment process that measures
  students' learning outcomes, AutoDetect consists of three LLM-powered agents:
  Examiner, Questioner, and Assessor. The collaboration among these three agents
  is designed to realize comprehensive and in-depth weakness identification. Our
  framework demonstrates significant success in uncovering flaws, with an
  identification success rate exceeding 30% in prominent models such as ChatGPT
  and Claude. More importantly, these identified weaknesses can guide specific
  model improvements, proving more effective than untargeted data augmentation
  methods like Self-Instruct. Our approach has led to substantial enhancements in
  popular LLMs, including the Llama series and Mistral-7b, boosting their
  performance by over 10% across several benchmarks. Code and data are publicly
  available at https://github.com/thu-coai/AutoDetect.
paper_ai_summary: A unified framework, AutoDetect, uses LLM-powered agents to systematically identify and address weaknesses in LLMs, improving their performance across benchmarks.
paper_ai_keywords: Large Language Models (LLMs) instruction-following coding tasks systematic limitations investigation traditional benchmarking manual inspections Examiner Questioner Assessor specific model deficiencies identification success rate Self-Instruct model improvements Llama series Mistral-7b