MoAI: Mixture of All Intelligence for Large Language and Vision Models
The rise of large language models (LLMs) and instruction tuning has led to
the current trend of instruction-tuned large language and vision models
(LLVMs). This trend involves either meticulously curating numerous instruction
tuning datasets tailored to specific objectives or enlarging LLVMs to manage
vast amounts of vision language (VL) data. However, current LLVMs have
disregarded the detailed and comprehensive real-world scene understanding
available from specialized computer vision (CV) models in visual perception
tasks such as segmentation, detection, scene graph generation (SGG), and
optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
the large capacity and emergent capabilities of their LLM backbones. Therefore,
we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
auxiliary visual information obtained from the outputs of external
segmentation, detection, SGG, and OCR models. MoAI operates through two newly
introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
outputs of the external CV models, the MoAI-Compressor aligns and condenses
them to efficiently use relevant auxiliary visual information for VL tasks.
MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
auxiliary features from the external CV models, and (3) language features by
utilizing the concept of Mixture of Experts. Through this integration, MoAI
significantly outperforms both open-source and closed-source LLVMs in numerous
zero-shot VL tasks, particularly those related to real-world scene
understanding such as object existence, positions, relations, and OCR without
enlarging the model size or curating extra visual instruction tuning datasets.
65f11a8f271175e4414b58ce
Byung-Kwan Lee
admin_assigned
2024-03-13T13:25:32.972Z
657152eb12f162153b50ec9d
https://cdn-avatars.huggingface.co/v1/production/uploads/657152eb12f162153b50ec9d/qnldHP35PclV0pDz_05q8.jpeg
Byung-Kwan Lee
user
BK-Lee
65f11a8f271175e4414b58cf
Beomchan Park
admin_assigned
2024-03-13T13:27:43.001Z
65390e7ce279d2fc80a3f800
/avatars/a930c1f20fa1eef372ca92139cb32887.svg
Beomchan Park
user
bpbpbp0810
65f11a8f271175e4414b58d0
Chae Won Kim
claimed_verified
2024-03-13T13:26:29.502Z
65aa2e84e2a2c86356263340
/avatars/cba9ecceff0f67b2b5dde02e7c0a9991.svg
Chae Won Kim
user
chae-won-kim
65f11a8f271175e4414b58d1
Yong Man Ro
claimed_verified
2024-04-02T11:04:32.843Z
660603529e3555d648e3baf1
/avatars/0f54479afcfc19df00b25d5aedb4cf67.svg
Yong Man Ro
user
dwightro
MoAI: Mixture of All Intelligence for Large Language and Vision Models
The rise of large language models (LLMs) and instruction tuning has led to
the current trend of instruction-tuned large language and vision models
(LLVMs). This trend involves either meticulously curating numerous instruction
tuning datasets tailored to specific objectives or enlarging LLVMs to manage
vast amounts of vision language (VL) data. However, current LLVMs have
disregarded the detailed and comprehensive real-world scene understanding
available from specialized computer vision (CV) models in visual perception
tasks such as segmentation, detection, scene graph generation (SGG), and
optical character recognition (OCR). Instead, the existing LLVMs rely mainly on
the large capacity and emergent capabilities of their LLM backbones. Therefore,
we present a new LLVM, Mixture of All Intelligence (MoAI), which leverages
auxiliary visual information obtained from the outputs of external
segmentation, detection, SGG, and OCR models. MoAI operates through two newly
introduced modules: MoAI-Compressor and MoAI-Mixer. After verbalizing the
outputs of the external CV models, the MoAI-Compressor aligns and condenses
them to efficiently use relevant auxiliary visual information for VL tasks.
MoAI-Mixer then blends three types of intelligence (1) visual features, (2)
auxiliary features from the external CV models, and (3) language features by
utilizing the concept of Mixture of Experts. Through this integration, MoAI
significantly outperforms both open-source and closed-source LLVMs in numerous
zero-shot VL tasks, particularly those related to real-world scene
understanding such as object existence, positions, relations, and OCR without
enlarging the model size or curating extra visual instruction tuning datasets.
MoAI, a new LLVM, integrates outputs from external CV models to enhance VL tasks without expanding model size or curating extra datasets.
LLMs instruction tuning LLVMs computer vision (CV) segmentation detection scene graph generation (SGG) optical character recognition (OCR) MoAI MoAI-Compressor MoAI-Mixer Mixture of Experts