title: Matryoshka Multimodal Models
summary: Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in
  visual-linguistic reasoning. These models first embed images into a fixed large
  number of visual tokens and then feed them into a Large Language Model (LLM).
  However, this design causes an excessive number of tokens for dense visual
  scenarios such as high-resolution images and videos, leading to great
  inefficiency. While token pruning/merging methods do exist, they produce a
  single length output for each image and do not afford flexibility in trading
  off information density v.s. efficiency. Inspired by the concept of Matryoshka
  Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent
  visual content as nested sets of visual tokens that capture information across
  multiple coarse-to-fine granularities. Our approach offers several unique
  benefits for LMMs: (1) One can explicitly control the visual granularity per
  test instance during inference, e.g. , adjusting the number of tokens used to
  represent an image based on the anticipated complexity or simplicity of the
  content; (2) M3 provides a framework for analyzing the granularity needed for
  existing datasets, where we find that COCO-style benchmarks only need around ~9
  visual tokens to obtain accuracy similar to that of using all 576 tokens; (3)
  Our approach provides a foundation to explore the best trade-off between
  performance and visual token length at sample level, where our investigation
  reveals that a large gap exists between the oracle upper bound and current
  fixed-scale representations.
paper_authors._id: 665547a7e6ddaa4f89ad2020
paper_authors.name: Mu Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T07:46:24.511Z
paper_authors._id: 63b7b2c6bd2d153522821766
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63b7b2c6bd2d153522821766/aHtga-_OUdOrg_TRrXO08.jpeg
paper_authors.fullname: Mu Cai
paper_authors.type: user
paper_authors.user: mucai
paper_authors._id: 665547a7e6ddaa4f89ad2021
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T08:12:03.879Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 665547a7e6ddaa4f89ad2022
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T08:12:12.457Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 665547a7e6ddaa4f89ad2023
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-28T08:12:18.996Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_title: Matryoshka Multimodal Models
paper_summary: Large Multimodal Models (LMMs) such as LLaVA have shown strong performance in
  visual-linguistic reasoning. These models first embed images into a fixed large
  number of visual tokens and then feed them into a Large Language Model (LLM).
  However, this design causes an excessive number of tokens for dense visual
  scenarios such as high-resolution images and videos, leading to great
  inefficiency. While token pruning/merging methods do exist, they produce a
  single length output for each image and do not afford flexibility in trading
  off information density v.s. efficiency. Inspired by the concept of Matryoshka
  Dolls, we propose M3: Matryoshka Multimodal Models, which learns to represent
  visual content as nested sets of visual tokens that capture information across
  multiple coarse-to-fine granularities. Our approach offers several unique
  benefits for LMMs: (1) One can explicitly control the visual granularity per
  test instance during inference, e.g. , adjusting the number of tokens used to
  represent an image based on the anticipated complexity or simplicity of the
  content; (2) M3 provides a framework for analyzing the granularity needed for
  existing datasets, where we find that COCO-style benchmarks only need around ~9
  visual tokens to obtain accuracy similar to that of using all 576 tokens; (3)
  Our approach provides a foundation to explore the best trade-off between
  performance and visual token length at sample level, where our investigation
  reveals that a large gap exists between the oracle upper bound and current
  fixed-scale representations.
paper_ai_keywords: Multimodal Models (LMMs) LLaVA visual-linguistic reasoning visual tokens Large Language Model (LLM) token pruning token merging Matryoshka Dolls Matryoshka Multimodal Models (M3) nested sets of visual tokens coarse-to-fine granularities visual granularity inference COPYO-style benchmarks