Mindstorms in Natural Language-Based Societies of Mind
Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire
diverse societies of large multimodal neural networks (NNs) that solve problems
by interviewing each other in a "mindstorm." Recent implementations of NN-based
societies of minds consist of large language models (LLMs) and other NN-based
experts communicating through a natural language interface. In doing so, they
overcome the limitations of single LLMs, improving multimodal zero-shot
reasoning. In these natural language-based societies of mind (NLSOMs), new
agents -- all communicating through the same universal symbolic language -- are
easily added in a modular fashion. To demonstrate the power of NLSOMs, we
assemble and experiment with several of them (having up to 129 members),
leveraging mindstorms in them to solve some practical AI tasks: visual question
answering, image captioning, text-to-image synthesis, 3D generation, egocentric
retrieval, embodied AI, and general language-based task solving. We view this
as a starting point towards much larger NLSOMs with billions of agents-some of
which may be humans. And with this emergence of great societies of
heterogeneous minds, many new research questions have suddenly become paramount
to the future of artificial intelligence. What should be the social structure
of an NLSOM? What would be the (dis)advantages of having a monarchical rather
than a democratic structure? How can principles of NN economies be used to
maximize the total reward of a reinforcement learning NLSOM? In this work, we
identify, discuss, and try to answer some of these questions.
64741754d9e9598973cce0aa
Mingchen Zhuge
claimed_verified
2025-04-04T07:26:21.743Z
64403daae44f30a72323e4ca
https://cdn-avatars.huggingface.co/v1/production/uploads/64403daae44f30a72323e4ca/skJ9h0pdNfmE4VbQL8xDR.png
mingchen zhuge
user
tjpxiaoming
64741754d9e9598973cce0ab
Haozhe Liu
64741754d9e9598973cce0ac
Francesco Faccio
64741754d9e9598973cce0ad
Dylan R. Ashley
64741754d9e9598973cce0ae
Róbert Csordás
claimed_verified
2025-03-11T08:26:22.042Z
64e5d1bef450dca5ca0d05ed
https://cdn-avatars.huggingface.co/v1/production/uploads/64e5d1bef450dca5ca0d05ed/HQHb6jM-3mmfCOzR-4eeW.jpeg
Robert Csordas
user
robertcsordas
64741754d9e9598973cce0af
Anand Gopalakrishnan
64741754d9e9598973cce0b0
Abdullah Hamdi
claimed_verified
2023-05-30T14:15:37.910Z
62fe3442e9061c0170d06e0b
https://cdn-avatars.huggingface.co/v1/production/uploads/1660827186084-62fe3442e9061c0170d06e0b.png
Abdullah Hamdi
user
ajhamdi
64741754d9e9598973cce0b1
Hasan Abed Al Kader Hammoud
claimed_verified
2023-05-30T17:59:59.408Z
642b51385bf2355d02a23d15
/avatars/87985347643b2647555f2453fa4d94fb.svg
Hasan Abed Al Kader Hammoud
user
hammh0a
64741754d9e9598973cce0b2
Vincent Herrmann
64741754d9e9598973cce0b3
Kazuki Irie
64741754d9e9598973cce0b4
Louis Kirsch
64741754d9e9598973cce0b5
Bing Li
claimed_verified
2025-03-31T08:18:03.615Z
666ddb45c0f3d5afc27e85ba
/avatars/dce98fc77bd8cf9e348f2d91bc3c0225.svg
Bing Li
user
bing-li-ai
64741754d9e9598973cce0b6
Guohao Li
64741754d9e9598973cce0b7
Shuming Liu
64741754d9e9598973cce0b8
Jinjie Mai
64741754d9e9598973cce0b9
Piotr Piękos
64741754d9e9598973cce0ba
Aditya Ramesh
64741754d9e9598973cce0bb
Imanol Schlag
64741754d9e9598973cce0bc
Weimin Shi
64741754d9e9598973cce0bd
Aleksandar Stanić
64741754d9e9598973cce0be
Wenyi Wang
64741754d9e9598973cce0bf
Yuhui Wang
64741754d9e9598973cce0c0
Mengmeng Xu
64741754d9e9598973cce0c1
Deng-Ping Fan
64741754d9e9598973cce0c2
Bernard Ghanem
64741754d9e9598973cce0c3
Jürgen Schmidhuber
Mindstorms in Natural Language-Based Societies of Mind
Both Minsky's "society of mind" and Schmidhuber's "learning to think" inspire
diverse societies of large multimodal neural networks (NNs) that solve problems
by interviewing each other in a "mindstorm." Recent implementations of NN-based
societies of minds consist of large language models (LLMs) and other NN-based
experts communicating through a natural language interface. In doing so, they
overcome the limitations of single LLMs, improving multimodal zero-shot
reasoning. In these natural language-based societies of mind (NLSOMs), new
agents -- all communicating through the same universal symbolic language -- are
easily added in a modular fashion. To demonstrate the power of NLSOMs, we
assemble and experiment with several of them (having up to 129 members),
leveraging mindstorms in them to solve some practical AI tasks: visual question
answering, image captioning, text-to-image synthesis, 3D generation, egocentric
retrieval, embodied AI, and general language-based task solving. We view this
as a starting point towards much larger NLSOMs with billions of agents-some of
which may be humans. And with this emergence of great societies of
heterogeneous minds, many new research questions have suddenly become paramount
to the future of artificial intelligence. What should be the social structure
of an NLSOM? What would be the (dis)advantages of having a monarchical rather
than a democratic structure? How can principles of NN economies be used to
maximize the total reward of a reinforcement learning NLSOM? In this work, we
identify, discuss, and try to answer some of these questions.
Natural language-based societies of minds (NLSOMs) consisting of diverse neural network experts improve multimodal zero-shot reasoning and can solve various practical AI tasks by communicating through a universal symbolic language.
society of mind learning to think neural networks large language models natural language interface multimodal zero-shot reasoning natural language-based societies of mind mindstorms visual question answering image captioning text-to-image synthesis 3D generation egocentric retrieval embodied AI reinforcement learning NN economies