title: MoAI: Mixture of All Intelligence for Large Language and Vision Models
summary: The rise of large language models (LLMs) and instruction tuning has led to
  the current trend of instruction-tuned large language and vision models
  (LLVMs). This trend involves either meticulously curating numerous instruction
  tuning datasets tailored to specific objectives or enlarging LLVMs to manage
  vast amounts of vision language (VL) data. However, current LLVMs have
  disregarded the detailed and comprehensive real-world scene understanding
  available from specialized computer vision (CV) models in visual perception
  tasks such as segmentation, detection, scene graph generation (SGG), and
  optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
  the large capacity and emergent capabilities of their LLM backbones. Therefore,
  we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
  auxiliary visual information obtained from the outputs of external
  segmentation, detection, SGG, and OCR models. MoAI operates through two newly
  introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
  outputs of the external CV models, the MoAI-Compressor aligns and condenses
  them to efficiently use relevant auxiliary visual information for VL tasks.
  MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
  auxiliary features from the external CV models, and (3) language features by
  utilizing the concept of Mixture of Experts. Through this integration, MoAI
  significantly outperforms both open-source and closed-source LLVMs in numerous
  zero-shot VL tasks, particularly those related to real-world scene
  understanding such as object existence, positions, relations, and OCR without
  enlarging the model size or curating extra visual instruction tuning datasets.
paper_authors._id: 65f11a8f271175e4414b58ce
paper_authors.name: Byung-Kwan Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:25:32.972Z
paper_authors._id: 657152eb12f162153b50ec9d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
paper_authors.fullname: Byung-Kwan Lee
paper_authors.type: user
paper_authors.user: BK-Lee
paper_authors._id: 65f11a8f271175e4414b58cf
paper_authors.name: Beomchan Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-13T13:27:43.001Z
paper_authors._id: 65390e7ce279d2fc80a3f800
paper_authors.avatarUrl: /avatars/a930c1f20fa1eef372ca92139cb32887.svg
paper_authors.fullname: Beomchan Park
paper_authors.type: user
paper_authors.user: bpbpbp0810
paper_authors._id: 65f11a8f271175e4414b58d0
paper_authors.name: Chae Won Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T13:26:29.502Z
paper_authors._id: 65aa2e84e2a2c86356263340
paper_authors.avatarUrl: /avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
paper_authors.fullname: Chae Won Kim
paper_authors.type: user
paper_authors.user: chae-won-kim
paper_authors._id: 65f11a8f271175e4414b58d1
paper_authors.name: Yong Man Ro
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-02T11:04:32.843Z
paper_authors._id: 660603529e3555d648e3baf1
paper_authors.avatarUrl: /avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
paper_authors.fullname: Yong Man Ro
paper_authors.type: user
paper_authors.user: dwightro
paper_title: MoAI: Mixture of All Intelligence for Large Language and Vision Models
paper_summary: The rise of large language models (LLMs) and instruction tuning has led to
  the current trend of instruction-tuned large language and vision models
  (LLVMs). This trend involves either meticulously curating numerous instruction
  tuning datasets tailored to specific objectives or enlarging LLVMs to manage
  vast amounts of vision language (VL) data. However, current LLVMs have
  disregarded the detailed and comprehensive real-world scene understanding
  available from specialized computer vision (CV) models in visual perception
  tasks such as segmentation, detection, scene graph generation (SGG), and
  optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
  the large capacity and emergent capabilities of their LLM backbones. Therefore,
  we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
  auxiliary visual information obtained from the outputs of external
  segmentation, detection, SGG, and OCR models. MoAI operates through two newly
  introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
  outputs of the external CV models, the MoAI-Compressor aligns and condenses
  them to efficiently use relevant auxiliary visual information for VL tasks.
  MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
  auxiliary features from the external CV models, and (3) language features by
  utilizing the concept of Mixture of Experts. Through this integration, MoAI
  significantly outperforms both open-source and closed-source LLVMs in numerous
  zero-shot VL tasks, particularly those related to real-world scene
  understanding such as object existence, positions, relations, and OCR without
  enlarging the model size or curating extra visual instruction tuning datasets.
paper_ai_keywords: instruction tuning large language models (LLMs) instruction-tuned large language and vision models (LLVMs) vision language (VL) data computer vision (CV) models segmentation detection scene graph generation (SGG) optical character recognition (OCR) Mixture of All Intelligence (MoAI) MoAI-Compressor MoAI-Mixer Mixture of Experts zero-shot VL tasks real-world scene understanding object existence positions relations