title: PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task
    Completion
summary: Recent evaluations of Large Language Models (LLMs) have centered around
  testing their zero-shot/few-shot capabilities for basic natural language tasks
  and their ability to translate instructions into tool APIs. However, the
  evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal
  instructions in a complex multi-modal environment has not been investigated. To
  address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark
  to assess LLMs' ability to create and edit PPT files based on user
  instructions. It contains 279 multi-turn sessions covering diverse topics and
  hundreds of instructions involving multi-modal operations. We also propose the
  PPTX-Match Evaluation System that evaluates if LLMs finish the instruction
  based on the prediction file rather than the label API sequence, thus it
  supports various LLM-generated API sequences. We measure 3 closed LLMs and 6
  open-source LLMs. The results show that GPT-4 outperforms other LLMs with
  75.1\% accuracy in single-turn dialogue testing but faces challenges in
  completing entire sessions, achieving just 6\% session accuracy. We find three
  main error causes in our benchmark: error accumulation in the multi-turn
  session, long PPT template processing, and multi-modality perception. These
  pose great challenges for future LLM and agent systems. We release the data,
  code, and evaluation system of PPTC at https://github.com/gydpku/PPTC.
paper_authors._id: 654874d85cd5692b3af9d354
paper_authors.name: Yiduo Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T13:59:37.823Z
paper_authors._id: 64e3282156b920ef00cf7d94
paper_authors.avatarUrl: /avatars/8a76c27fe6c71eb98a28b4ebfb90336d.svg
paper_authors.fullname: Guo
paper_authors.type: user
paper_authors.user: YiDuo1999
paper_authors._id: 654874d85cd5692b3af9d355
paper_authors.name: Zekai Zhang
paper_authors._id: 654874d85cd5692b3af9d356
paper_authors.name: Yaobo Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T14:00:19.782Z
paper_authors._id: 64ddde772402348182664c2a
paper_authors.avatarUrl: /avatars/21798dde3df06f57288dc9dc2e044abc.svg
paper_authors.fullname: Yaobo Liang
paper_authors.type: user
paper_authors.user: Eureka6174
paper_authors._id: 654874d85cd5692b3af9d357
paper_authors.name: Dongyan Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-06T14:01:22.213Z
paper_authors._id: 629ee8171fec321a2d3c458d
paper_authors.avatarUrl: /avatars/a8ae78b193ba2a43c6e36e185e0ef2b9.svg
paper_authors.fullname: Zhaodong Yan
paper_authors.type: user
paper_authors.user: Anhforth
paper_authors._id: 654874d85cd5692b3af9d358
paper_authors.name: Duan Nan
paper_title: PPTC Benchmark: Evaluating Large Language Models for PowerPoint Task
    Completion
paper_summary: Recent evaluations of Large Language Models (LLMs) have centered around
  testing their zero-shot/few-shot capabilities for basic natural language tasks
  and their ability to translate instructions into tool APIs. However, the
  evaluation of LLMs utilizing complex tools to finish multi-turn, multi-modal
  instructions in a complex multi-modal environment has not been investigated. To
  address this gap, we introduce the PowerPoint Task Completion (PPTC) benchmark
  to assess LLMs' ability to create and edit PPT files based on user
  instructions. It contains 279 multi-turn sessions covering diverse topics and
  hundreds of instructions involving multi-modal operations. We also propose the
  PPTX-Match Evaluation System that evaluates if LLMs finish the instruction
  based on the prediction file rather than the label API sequence, thus it
  supports various LLM-generated API sequences. We measure 3 closed LLMs and 6
  open-source LLMs. The results show that GPT-4 outperforms other LLMs with
  75.1\% accuracy in single-turn dialogue testing but faces challenges in
  completing entire sessions, achieving just 6\% session accuracy. We find three
  main error causes in our benchmark: error accumulation in the multi-turn
  session, long PPT template processing, and multi-modality perception. These
  pose great challenges for future LLM and agent systems. We release the data,
  code, and evaluation system of PPTC at https://github.com/gydpku/PPTC.
paper_ai_keywords: PowerPoint Task Completion (PPTC) LLMs (Large Language Models) zero-shot few-shot natural language tasks tool APIs multi-turn multi-modal PPT files multi-modal operations PPTX-Match Evaluation System API sequences GPT-4 error accumulation PPT template processing multi-modality perception