MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf
In contemporary workplaces, meetings are essential for exchanging ideas and
ensuring team alignment but often face challenges such as time consumption,
scheduling conflicts, and inefficient participation. Recent advancements in
Large Language Models (LLMs) have demonstrated their strong capabilities in
natural language generation and reasoning, prompting the question: can LLMs
effectively delegate participants in meetings? To explore this, we develop a
prototype LLM-powered meeting delegate system and create a comprehensive
benchmark using real meeting transcripts. Our evaluation reveals that GPT-4/4o
maintain balanced performance between active and cautious engagement
strategies. In contrast, Gemini 1.5 Pro tends to be more cautious, while Gemini
1.5 Flash and Llama3-8B/70B display more active tendencies. Overall, about 60\%
of responses address at least one key point from the ground-truth. However,
improvements are needed to reduce irrelevant or repetitive content and enhance
tolerance for transcription errors commonly found in real-world settings.
Additionally, we implement the system in practical settings and collect
real-world feedback from demos. Our findings underscore the potential and
challenges of utilizing LLMs as meeting delegates, offering valuable insights
into their practical application for alleviating the burden of meetings.
67a998fe495b23306cdbf51d
Lingxiang Hu
67a998fe495b23306cdbf51e
Shurun Yuan
67a998fe495b23306cdbf51f
Xiaoting Qin
67a998fe495b23306cdbf520
Jue Zhang
67a998fe495b23306cdbf521
Qingwei Lin
67a998fe495b23306cdbf522
Dongmei Zhang
67a998fe495b23306cdbf523
Saravan Rajmohan
67a998fe495b23306cdbf524
Qi Zhang
MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf
In contemporary workplaces, meetings are essential for exchanging ideas and
ensuring team alignment but often face challenges such as time consumption,
scheduling conflicts, and inefficient participation. Recent advancements in
Large Language Models (LLMs) have demonstrated their strong capabilities in
natural language generation and reasoning, prompting the question: can LLMs
effectively delegate participants in meetings? To explore this, we develop a
prototype LLM-powered meeting delegate system and create a comprehensive
benchmark using real meeting transcripts. Our evaluation reveals that GPT-4/4o
maintain balanced performance between active and cautious engagement
strategies. In contrast, Gemini 1.5 Pro tends to be more cautious, while Gemini
1.5 Flash and Llama3-8B/70B display more active tendencies. Overall, about 60\%
of responses address at least one key point from the ground-truth. However,
improvements are needed to reduce irrelevant or repetitive content and enhance
tolerance for transcription errors commonly found in real-world settings.
Additionally, we implement the system in practical settings and collect
real-world feedback from demos. Our findings underscore the potential and
challenges of utilizing LLMs as meeting delegates, offering valuable insights
into their practical application for alleviating the burden of meetings.
LLM-powered meeting delegate systems show potential in handling real meeting transcripts but need improvements in relevance and error tolerance.
Large Language Models LLM GPT-4/4o Gemini 1.5 Pro Gemini 1.5 Flash Llama3-8B/70B meeting delegate system natural language generation reasoning transcription errors