Diversity Empowers Intelligence: Integrating Expertise of Software
  Engineering Agents
Large language model (LLM) agents have shown great potential in solving
real-world software engineering (SWE) problems. The most advanced open-source
SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite.
However, these sophisticated agent frameworks exhibit varying strengths,
excelling in certain tasks while underperforming in others. To fully harness
the diversity of these agents, we propose DEI (Diversity Empowered
Intelligence), a framework that leverages their unique expertise. DEI functions
as a meta-module atop existing SWE agent frameworks, managing agent collectives
for enhanced problem-solving. Experimental results show that a DEI-guided
committee of agents is able to surpass the best individual agent's performance
by a large margin. For instance, a group of open-source SWE agents, with a
maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3%
resolve rate with DEI, making a 25% improvement and beating most closed-source
solutions. Our best-performing group excels with a 55% resolve rate, securing
the highest ranking on SWE-Bench Lite. Our findings contribute to the growing
body of research on collaborative AI systems and their potential to solve
complex software engineering challenges.
66bcd64b85c8bcb5f2681504
Kexun Zhang
claimed_verified
2024-08-15T10:11:03.448Z
62ee423b4bebb4ab55c674b1
/avatars/ce2797937e8225937fc84d6847d50077.svg
Kexun Zhang
user
k1z
66bcd64b85c8bcb5f2681505
Weiran Yao
extracted_pending
2024-08-14T16:07:39.897Z
661573234c2f29635e93bb71
/avatars/fba95e382454485766b6349d6281b715.svg
Weiran Yao
user
weiranyao
66bcd64b85c8bcb5f2681506
Zuxin Liu
claimed_verified
2024-08-15T10:09:35.758Z
638e971f2cc490759fe9210e
/avatars/9578d0badc084d340c028a59e571abd3.svg
Liu
user
Zuxin
66bcd64b85c8bcb5f2681507
Yihao Feng
66bcd64b85c8bcb5f2681508
Zhiwei Liu
66bcd64b85c8bcb5f2681509
Rithesh Murthy
claimed_verified
2024-08-15T10:09:33.543Z
60381555c58b41eeda0d6ccf
/avatars/2c2c4b75dd0199478dc0fbc72f9bcadc.svg
rmurthy
user
rmurthy
66bcd64b85c8bcb5f268150a
Tian Lan
66bcd64b85c8bcb5f268150b
Lei Li
66bcd64b85c8bcb5f268150c
Renze Lou
66bcd64b85c8bcb5f268150d
Jiacheng Xu
66bcd64b85c8bcb5f268150e
Bo Pang
66bcd64b85c8bcb5f268150f
Yingbo Zhou
66bcd64b85c8bcb5f2681510
Shelby Heinecke
66bcd64b85c8bcb5f2681511
Silvio Savarese
66bcd64b85c8bcb5f2681512
Huan Wang
66bcd64b85c8bcb5f2681513
Caiming Xiong
Diversity Empowers Intelligence: Integrating Expertise of Software
  Engineering Agents
Large language model (LLM) agents have shown great potential in solving
real-world software engineering (SWE) problems. The most advanced open-source
SWE agent can resolve over 27% of real GitHub issues in SWE-Bench Lite.
However, these sophisticated agent frameworks exhibit varying strengths,
excelling in certain tasks while underperforming in others. To fully harness
the diversity of these agents, we propose DEI (Diversity Empowered
Intelligence), a framework that leverages their unique expertise. DEI functions
as a meta-module atop existing SWE agent frameworks, managing agent collectives
for enhanced problem-solving. Experimental results show that a DEI-guided
committee of agents is able to surpass the best individual agent's performance
by a large margin. For instance, a group of open-source SWE agents, with a
maximum individual resolve rate of 27.3% on SWE-Bench Lite, can achieve a 34.3%
resolve rate with DEI, making a 25% improvement and beating most closed-source
solutions. Our best-performing group excels with a 55% resolve rate, securing
the highest ranking on SWE-Bench Lite. Our findings contribute to the growing
body of research on collaborative AI systems and their potential to solve
complex software engineering challenges.
A DEI framework enhances software engineering problem-solving by leveraging a collective of LLM agents, achieving higher resolve rates compared to individual agents and closed-source solutions.
LLM agents software engineering SWE agent frameworks DEI (Diversity Empowered Intelligence) meta-module agent collectives SWE-Bench Lite resolve rate collaborative AI systems