title: Mindstorms in Natural Language-Based Societies of Mind
summary: Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire
  diverse societies of large multimodal neural networks (NNs) that solve problems
  by interviewing each other in a "mindstorm." Recent implementations of NN-based
  societies of minds consist of large language models (LLMs) and other NN-based
  experts communicating through a natural language interface. In doing so, they
  overcome the limitations of single LLMs, improving multimodal zero-shot
  reasoning. In these natural language-based societies of mind (NLSOMs), new
  agents -- all communicating through the same universal symbolic language -- are
  easily added in a modular fashion. To demonstrate the power of NLSOMs, we
  assemble and experiment with several of them (having up to 129 members),
  leveraging mindstorms in them to solve some practical AI tasks: visual question
  answering, image captioning, text-to-image synthesis, 3D generation, egocentric
  retrieval, embodied AI, and general language-based task solving. We view this
  as a starting point towards much larger NLSOMs with billions of agents-some of
  which may be humans. And with this emergence of great societies of
  heterogeneous minds, many new research questions have suddenly become paramount
  to the future of artificial intelligence. What should be the social structure
  of an NLSOM? What would be the (dis)advantages of having a monarchical rather
  than a democratic structure? How can principles of NN economies be used to
  maximize the total reward of a reinforcement learning NLSOM? In this work, we
  identify, discuss, and try to answer some of these questions.
paper_authors._id: 64741754d9e9598973cce0aa
paper_authors.name: Mingchen Zhuge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:26:21.743Z
paper_authors._id: 64403daae44f30a72323e4ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64403daae44f30a72323e4ca/skJ9h0pdNfmE4VbQL8xDR.png
paper_authors.fullname: mingchen zhuge
paper_authors.type: user
paper_authors.user: tjpxiaoming
paper_authors._id: 64741754d9e9598973cce0ab
paper_authors.name: Haozhe Liu
paper_authors._id: 64741754d9e9598973cce0ac
paper_authors.name: Francesco Faccio
paper_authors._id: 64741754d9e9598973cce0ad
paper_authors.name: Dylan R. Ashley
paper_authors._id: 64741754d9e9598973cce0ae
paper_authors.name: Róbert Csordás
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T08:26:22.042Z
paper_authors._id: 64e5d1bef450dca5ca0d05ed
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
paper_authors.fullname: Robert Csordas
paper_authors.type: user
paper_authors.user: robertcsordas
paper_authors._id: 64741754d9e9598973cce0af
paper_authors.name: Anand Gopalakrishnan
paper_authors._id: 64741754d9e9598973cce0b0
paper_authors.name: Abdullah Hamdi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T14:15:37.910Z
paper_authors._id: 62fe3442e9061c0170d06e0b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
paper_authors.fullname: Abdullah Hamdi
paper_authors.type: user
paper_authors.user: ajhamdi
paper_authors._id: 64741754d9e9598973cce0b1
paper_authors.name: Hasan Abed Al Kader Hammoud
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-30T17:59:59.408Z
paper_authors._id: 642b51385bf2355d02a23d15
paper_authors.avatarUrl: /avatars/87985347643b2647555f2453fa4d94fb.svg
paper_authors.fullname: Hasan Abed Al Kader Hammoud
paper_authors.type: user
paper_authors.user: hammh0a
paper_authors._id: 64741754d9e9598973cce0b2
paper_authors.name: Vincent Herrmann
paper_authors._id: 64741754d9e9598973cce0b3
paper_authors.name: Kazuki Irie
paper_authors._id: 64741754d9e9598973cce0b4
paper_authors.name: Louis Kirsch
paper_authors._id: 64741754d9e9598973cce0b5
paper_authors.name: Bing Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:18:03.615Z
paper_authors._id: 666ddb45c0f3d5afc27e85ba
paper_authors.avatarUrl: /avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
paper_authors.fullname: Bing Li
paper_authors.type: user
paper_authors.user: bing-li-ai
paper_authors._id: 64741754d9e9598973cce0b6
paper_authors.name: Guohao Li
paper_authors._id: 64741754d9e9598973cce0b7
paper_authors.name: Shuming Liu
paper_authors._id: 64741754d9e9598973cce0b8
paper_authors.name: Jinjie Mai
paper_authors._id: 64741754d9e9598973cce0b9
paper_authors.name: Piotr Piękos
paper_authors._id: 64741754d9e9598973cce0ba
paper_authors.name: Aditya Ramesh
paper_authors._id: 64741754d9e9598973cce0bb
paper_authors.name: Imanol Schlag
paper_authors._id: 64741754d9e9598973cce0bc
paper_authors.name: Weimin Shi
paper_authors._id: 64741754d9e9598973cce0bd
paper_authors.name: Aleksandar Stanić
paper_authors._id: 64741754d9e9598973cce0be
paper_authors.name: Wenyi Wang
paper_authors._id: 64741754d9e9598973cce0bf
paper_authors.name: Yuhui Wang
paper_authors._id: 64741754d9e9598973cce0c0
paper_authors.name: Mengmeng Xu
paper_authors._id: 64741754d9e9598973cce0c1
paper_authors.name: Deng-Ping Fan
paper_authors._id: 64741754d9e9598973cce0c2
paper_authors.name: Bernard Ghanem
paper_authors._id: 64741754d9e9598973cce0c3
paper_authors.name: Jürgen Schmidhuber
paper_title: Mindstorms in Natural Language-Based Societies of Mind
paper_summary: Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire
  diverse societies of large multimodal neural networks (NNs) that solve problems
  by interviewing each other in a "mindstorm." Recent implementations of NN-based
  societies of minds consist of large language models (LLMs) and other NN-based
  experts communicating through a natural language interface. In doing so, they
  overcome the limitations of single LLMs, improving multimodal zero-shot
  reasoning. In these natural language-based societies of mind (NLSOMs), new
  agents -- all communicating through the same universal symbolic language -- are
  easily added in a modular fashion. To demonstrate the power of NLSOMs, we
  assemble and experiment with several of them (having up to 129 members),
  leveraging mindstorms in them to solve some practical AI tasks: visual question
  answering, image captioning, text-to-image synthesis, 3D generation, egocentric
  retrieval, embodied AI, and general language-based task solving. We view this
  as a starting point towards much larger NLSOMs with billions of agents-some of
  which may be humans. And with this emergence of great societies of
  heterogeneous minds, many new research questions have suddenly become paramount
  to the future of artificial intelligence. What should be the social structure
  of an NLSOM? What would be the (dis)advantages of having a monarchical rather
  than a democratic structure? How can principles of NN economies be used to
  maximize the total reward of a reinforcement learning NLSOM? In this work, we
  identify, discuss, and try to answer some of these questions.
paper_ai_summary: Natural language-based societies of minds (NLSOMs) consisting of diverse neural network experts improve multimodal zero-shot reasoning and can solve various practical AI tasks by communicating through a universal symbolic language.
paper_ai_keywords: society of mind learning to think neural networks large language models natural language interface multimodal zero-shot reasoning natural language-based societies of mind mindstorms visual question answering image captioning text-to-image synthesis 3D generation egocentric retrieval embodied AI reinforcement learning NN economies