title: Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
    Text Integration
summary: Although instruction-tuned large language models (LLMs) have exhibited
  remarkable capabilities across various NLP tasks, their effectiveness on other
  data modalities beyond text has not been fully studied. In this work, we
  propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
  audio, and textual information. Macaw-LLM consists of three main components: a
  modality module for encoding multi-modal data, a cognitive module for
  harnessing pretrained LLMs, and an alignment module for harmonizing diverse
  representations. Our novel alignment module seamlessly bridges multi-modal
  features to textual features, simplifying the adaptation process from the
  modality modules to the cognitive module. In addition, we construct a
  large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
  including 69K image instances and 50K video instances. We have made our data,
  code and model publicly available, which we hope can pave the way for future
  research in multi-modal LLMs and expand the capabilities of LLMs to handle
  diverse data modalities and address complex real-world scenarios.
paper_authors._id: 648bd92a9f4df9495b8adcb7
paper_authors.name: Chenyang Lyu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-23T08:41:07.726Z
paper_authors._id: 6527d8b077bceabaab382a75
paper_authors.avatarUrl: /avatars/69caacf9153dbf6a3796693a968b363f.svg
paper_authors.fullname: Chenyang Lyu
paper_authors.type: user
paper_authors.user: ChenyangLyu
paper_authors._id: 648bd92a9f4df9495b8adcb8
paper_authors.name: Minghao Wu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-06-16T03:44:40.608Z
paper_authors._id: 62d4bf8c97ab9eb08762a975
paper_authors.avatarUrl: /avatars/73c6228e317cf37b4e3c3e7a4b3d8ae8.svg
paper_authors.fullname: Minghao Wu
paper_authors.type: user
paper_authors.user: minghaowu
paper_authors._id: 648bd92a9f4df9495b8adcb9
paper_authors.name: Longyue Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:59:44.116Z
paper_authors._id: 636b030c328133bdb3a523bc
paper_authors.avatarUrl: /avatars/15d5d5403fef2f1368bb4185b199061d.svg
paper_authors.fullname: Longyue Wang
paper_authors.type: user
paper_authors.user: longyuewang
paper_authors._id: 648bd92a9f4df9495b8adcba
paper_authors.name: Xinting Huang
paper_authors._id: 648bd92a9f4df9495b8adcbb
paper_authors.name: Bingshuai Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:59:35.636Z
paper_authors._id: 6347f4968af48bc2fb5663f8
paper_authors.avatarUrl: /avatars/3941abd3cfdb68212db6106e2ce521ed.svg
paper_authors.fullname: Bingshuai Liu
paper_authors.type: user
paper_authors.user: bsliu
paper_authors._id: 648bd92a9f4df9495b8adcbc
paper_authors.name: Zefeng Du
paper_authors._id: 648bd92a9f4df9495b8adcbd
paper_authors.name: Shuming Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T12:00:32.800Z
paper_authors._id: 62eb8e9eaf9bed70f5ad5a7e
paper_authors.avatarUrl: /avatars/971763f3b724d63bc64b7d3599cfc753.svg
paper_authors.fullname: Shuming Shi
paper_authors.type: user
paper_authors.user: Shuming
paper_authors._id: 648bd92a9f4df9495b8adcbe
paper_authors.name: Zhaopeng Tu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T11:44:29.528Z
paper_authors._id: 67485743561b1e6f9579389f
paper_authors.avatarUrl: /avatars/8a4cc63bd7be388010bc329bb74582a1.svg
paper_authors.fullname: Zhaopeng Tu
paper_authors.type: user
paper_authors.user: zptu
paper_title: Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
    Text Integration
paper_summary: Although instruction-tuned large language models (LLMs) have exhibited
  remarkable capabilities across various NLP tasks, their effectiveness on other
  data modalities beyond text has not been fully studied. In this work, we
  propose Macaw-LLM, a novel multi-modal LLM that seamlessly integrates visual,
  audio, and textual information. Macaw-LLM consists of three main components: a
  modality module for encoding multi-modal data, a cognitive module for
  harnessing pretrained LLMs, and an alignment module for harmonizing diverse
  representations. Our novel alignment module seamlessly bridges multi-modal
  features to textual features, simplifying the adaptation process from the
  modality modules to the cognitive module. In addition, we construct a
  large-scale multi-modal instruction dataset in terms of multi-turn dialogue,
  including 69K image instances and 50K video instances. We have made our data,
  code and model publicly available, which we hope can pave the way for future
  research in multi-modal LLMs and expand the capabilities of LLMs to handle
  diverse data modalities and address complex real-world scenarios.
paper_ai_summary: Macaw-LLM integrates multimodal data including visual, audio, and text by employing a modality module, cognitive module, and novel alignment module, enhancing LLM capabilities across diverse data types.
paper_ai_keywords: instruction-tuned large language models multi-modal LLM modality module cognitive module alignment module multi-turn dialogue image instances video instances