title: AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
summary: We present Any-Modality Augmented Language Model (AnyMAL), a unified model
  that reasons over diverse input modality signals (i.e. text, image, video,
  audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
  powerful text-based reasoning abilities of the state-of-the-art LLMs including
  LLaMA-2 (70B), and converts modality-specific signals to the joint textual
  space through a pre-trained aligner module. To further strengthen the
  multimodal LLM's capabilities, we fine-tune the model with a multimodal
  instruction set manually collected to cover diverse topics and tasks beyond
  simple QAs. We conduct comprehensive empirical analysis comprising both human
  and automatic evaluations, and demonstrate state-of-the-art performance on
  various multimodal tasks.
paper_authors._id: 65162bcf3e5a12e0ceb42dfe
paper_authors.name: Seungwhan Moon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:18:51.029Z
paper_authors._id: 64c9ae24e761f47061448bfb
paper_authors.avatarUrl: /avatars/8835bcf769fd738e8c66fa7a52ddc421.svg
paper_authors.fullname: Seungwhan Moon
paper_authors.type: user
paper_authors.user: shanemoon27
paper_authors._id: 65162bcf3e5a12e0ceb42dff
paper_authors.name: Andrea Madotto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:18:58.812Z
paper_authors._id: 609bc5719da7ce3ca8930b8e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1620821302901-noauth.jpeg
paper_authors.fullname: Andrea Madotto
paper_authors.type: user
paper_authors.user: andreamad8
paper_authors._id: 65162bcf3e5a12e0ceb42e00
paper_authors.name: Zhaojiang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:19:05.658Z
paper_authors._id: 64b6d095dbbd1f2cdb62a946
paper_authors.avatarUrl: /avatars/b5f97735045d05ba5d30ee1fbf3f9bae.svg
paper_authors.fullname: Zhaojiang Lin
paper_authors.type: user
paper_authors.user: zlinao
paper_authors._id: 65162bcf3e5a12e0ceb42e01
paper_authors.name: Tushar Nagarajan
paper_authors._id: 65162bcf3e5a12e0ceb42e02
paper_authors.name: Matt Smith
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-04T07:37:45.913Z
paper_authors._id: 630872613485f377829de16f
paper_authors.avatarUrl: /avatars/412d46da44f357b549d58e82d789d98d.svg
paper_authors.fullname: Matt Smith
paper_authors.type: user
paper_authors.user: mjksmith
paper_authors._id: 65162bcf3e5a12e0ceb42e03
paper_authors.name: Shashank Jain
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:20:04.248Z
paper_authors._id: 64b6bdc461dc301f13230172
paper_authors.avatarUrl: /avatars/f43bb430c1c5508c6b52bdd9c0a8bfae.svg
paper_authors.fullname: Shashank Jain
paper_authors.type: user
paper_authors.user: shashankneocfc
paper_authors._id: 65162bcf3e5a12e0ceb42e04
paper_authors.name: Chun-Fu Yeh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-29T11:20:13.048Z
paper_authors._id: 64110236230ce11b1be5f869
paper_authors.avatarUrl: /avatars/f8718d21f7ea8b908afa938233a77f83.svg
paper_authors.fullname: Chun-Fu Yeh
paper_authors.type: user
paper_authors.user: cfyeh
paper_authors._id: 65162bcf3e5a12e0ceb42e05
paper_authors.name: Prakash Murugesan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-10T07:58:25.576Z
paper_authors._id: 654dd8a10011607c3125f107
paper_authors.avatarUrl: /avatars/9235811dd394c08116ceed32d943e968.svg
paper_authors.fullname: Prakash Murugesan
paper_authors.type: user
paper_authors.user: prakash14
paper_authors._id: 65162bcf3e5a12e0ceb42e06
paper_authors.name: Peyman Heidari
paper_authors._id: 65162bcf3e5a12e0ceb42e07
paper_authors.name: Yue Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-10T17:09:07.936Z
paper_authors._id: 66671b27ad9cd05ea74202a5
paper_authors.avatarUrl: /avatars/a6fa26c173f3e5b0748e5fb007c3105e.svg
paper_authors.fullname: Yue Liu
paper_authors.type: user
paper_authors.user: liu-yue
paper_authors._id: 65162bcf3e5a12e0ceb42e08
paper_authors.name: Kavya Srinet
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-04T08:03:36.763Z
paper_authors._id: 656a27e43eb5f0b6a9bef54a
paper_authors.avatarUrl: /avatars/32176455b54496917152b009e03efc2a.svg
paper_authors.fullname: Kavya Srinet
paper_authors.type: user
paper_authors.user: kavyasrinet
paper_authors._id: 65162bcf3e5a12e0ceb42e09
paper_authors.name: Babak Damavandi
paper_authors._id: 65162bcf3e5a12e0ceb42e0a
paper_authors.name: Anuj Kumar
paper_title: AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
paper_summary: We present Any-Modality Augmented Language Model (AnyMAL), a unified model
  that reasons over diverse input modality signals (i.e. text, image, video,
  audio, IMU motion sensor), and generates textual responses. AnyMAL inherits the
  powerful text-based reasoning abilities of the state-of-the-art LLMs including
  LLaMA-2 (70B), and converts modality-specific signals to the joint textual
  space through a pre-trained aligner module. To further strengthen the
  multimodal LLM's capabilities, we fine-tune the model with a multimodal
  instruction set manually collected to cover diverse topics and tasks beyond
  simple QAs. We conduct comprehensive empirical analysis comprising both human
  and automatic evaluations, and demonstrate state-of-the-art performance on
  various multimodal tasks.
paper_ai_keywords: Any-Modality Augmented Language Model (AnyMAL) input modality signals LLaMA-2 pre-trained aligner module multimodal instruction set multimodal LLM comprehensive empirical analysis human evaluations automatic evaluations state-of-the-art performance multimodal tasks