title: MindAgent: Emergent Gaming Interaction
summary: Large Language Models (LLMs) have the capacity of performing complex
  scheduling in a multi-agent system and can coordinate these agents into
  completing sophisticated tasks that require extensive collaboration. However,
  despite the introduction of numerous gaming frameworks, the community has
  insufficient benchmarks towards building general multi-agents collaboration
  infrastructure that encompass both LLM and human-NPCs collaborations. In this
  work, we propose a novel infrastructure - MindAgent - to evaluate planning and
  coordination emergent capabilities for gaming interaction. In particular, our
  infrastructure leverages existing gaming framework, to i) require understanding
  of the coordinator for a multi-agent system, ii) collaborate with human players
  via un-finetuned proper instructions, and iii) establish an in-context learning
  on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new
  gaming scenario and related benchmark that dispatch a multi-agent collaboration
  efficiency and supervise multiple agents playing the game simultaneously. We
  conduct comprehensive evaluations with new auto-metric CoS for calculating the
  collaboration efficiency. Finally, our infrastructure can be deployed into
  real-world gaming scenarios in a customized VR version of CUISINEWORLD and
  adapted in existing broader Minecraft gaming domain. We hope our findings on
  LLMs and the new infrastructure for general-purpose scheduling and coordination
  can help shed light on how such skills can be obtained by learning from large
  language corpora.
paper_authors._id: 650914e6948ce5dce89c8b81
paper_authors.name: Ran Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:24:15.670Z
paper_authors._id: 64acdaf75c6fc404f0d95e5b
paper_authors.avatarUrl: /avatars/a4235fb6984e5945eb0e11c3ce581b59.svg
paper_authors.fullname: Ran Gong
paper_authors.type: user
paper_authors.user: nikepupu
paper_authors._id: 650914e6948ce5dce89c8b82
paper_authors.name: Qiuyuan Huang
paper_authors._id: 650914e6948ce5dce89c8b83
paper_authors.name: Xiaojian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:24:38.100Z
paper_authors._id: 60dd0e36a15ddd7d2006d2e9
paper_authors.avatarUrl: /avatars/8bd98177a79efbf295be8f6457683297.svg
paper_authors.fullname: Xiaojian Ma
paper_authors.type: user
paper_authors.user: jeasinema
paper_authors._id: 650914e6948ce5dce89c8b84
paper_authors.name: Hoi Vo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:24:47.429Z
paper_authors._id: 631aae9bfac58c9c816636e1
paper_authors.avatarUrl: /avatars/92d030f9eb552a334aa129e02d3f2ca6.svg
paper_authors.fullname: Hoi Vo
paper_authors.type: user
paper_authors.user: Hoivo63
paper_authors._id: 650914e6948ce5dce89c8b85
paper_authors.name: Zane Durante
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:24:54.528Z
paper_authors._id: 650259fda33ce3ae28620bf3
paper_authors.avatarUrl: /avatars/5fb455b2e3a9523f07230d81d1947e15.svg
paper_authors.fullname: Zane Durante
paper_authors.type: user
paper_authors.user: ZaneDurante
paper_authors._id: 650914e6948ce5dce89c8b86
paper_authors.name: Yusuke Noda
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:25:02.750Z
paper_authors._id: 6430f78767688e5e12a408e4
paper_authors.avatarUrl: /avatars/a7a10d30123495a3eb87d944ae9f2b55.svg
paper_authors.fullname: Yusuke Noda
paper_authors.type: user
paper_authors.user: yusukenoda
paper_authors._id: 650914e6948ce5dce89c8b87
paper_authors.name: Zilong Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:25:09.855Z
paper_authors._id: 63a95a6a7930fa8c7dd63d4e
paper_authors.avatarUrl: /avatars/d9d0420f7ddfe2f3a7e029fb05f1c89f.svg
paper_authors.fullname: Zilong Zheng
paper_authors.type: user
paper_authors.user: zlzheng
paper_authors._id: 650914e6948ce5dce89c8b88
paper_authors.name: Song-Chun Zhu
paper_authors._id: 650914e6948ce5dce89c8b89
paper_authors.name: Demetri Terzopoulos
paper_authors._id: 650914e6948ce5dce89c8b8a
paper_authors.name: Li Fei-Fei
paper_authors._id: 650914e6948ce5dce89c8b8b
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-20T14:26:08.207Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_title: MindAgent: Emergent Gaming Interaction
paper_summary: Large Language Models (LLMs) have the capacity of performing complex
  scheduling in a multi-agent system and can coordinate these agents into
  completing sophisticated tasks that require extensive collaboration. However,
  despite the introduction of numerous gaming frameworks, the community has
  insufficient benchmarks towards building general multi-agents collaboration
  infrastructure that encompass both LLM and human-NPCs collaborations. In this
  work, we propose a novel infrastructure - MindAgent - to evaluate planning and
  coordination emergent capabilities for gaming interaction. In particular, our
  infrastructure leverages existing gaming framework, to i) require understanding
  of the coordinator for a multi-agent system, ii) collaborate with human players
  via un-finetuned proper instructions, and iii) establish an in-context learning
  on few-shot prompt with feedback. Furthermore, we introduce CUISINEWORLD, a new
  gaming scenario and related benchmark that dispatch a multi-agent collaboration
  efficiency and supervise multiple agents playing the game simultaneously. We
  conduct comprehensive evaluations with new auto-metric CoS for calculating the
  collaboration efficiency. Finally, our infrastructure can be deployed into
  real-world gaming scenarios in a customized VR version of CUISINEWORLD and
  adapted in existing broader Minecraft gaming domain. We hope our findings on
  LLMs and the new infrastructure for general-purpose scheduling and coordination
  can help shed light on how such skills can be obtained by learning from large
  language corpora.
paper_ai_keywords: MindAgent CUISINEWORLD in-context learning few-shot prompt feedback coordination efficiency CoS VR version Minecraft gaming domain