title: Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM
summary: Large multimodal models excel in multimodal tasks but face significant
  computational challenges due to excessive computation on visual tokens. Unlike
  token reduction methods that focus on token-level redundancy, we identify and
  study the computation-level redundancy on vision tokens to ensure no
  information loss. Our key insight is that vision tokens from the pretrained
  vision encoder do not necessarily require all the heavy operations (e.g.,
  self-attention, FFNs) in decoder-only LMMs and could be processed more lightly
  with proper designs. We designed a series of experiments to discover and
  progressively squeeze out the vision-related computation redundancy. Based on
  our findings, we propose ProxyV, a novel approach that utilizes proxy vision
  tokens to alleviate the computational burden on original vision tokens. ProxyV
  enhances efficiency without compromising performance and can even yield notable
  performance gains in scenarios with more moderate efficiency improvements.
  Furthermore, the flexibility of ProxyV is demonstrated through its combination
  with token reduction methods to boost efficiency further. The code will be made
  public at this https://github.com/penghao-wu/ProxyV URL.
paper_authors._id: 682e8304e9980508c9a2fc73
paper_authors.name: Penghao Wu
paper_authors._id: 682e8304e9980508c9a2fc74
paper_authors.name: Lewei Lu
paper_authors._id: 682e8304e9980508c9a2fc75
paper_authors.name: Ziwei Liu
paper_title: Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM
paper_summary: Large multimodal models excel in multimodal tasks but face significant
  computational challenges due to excessive computation on visual tokens. Unlike
  token reduction methods that focus on token-level redundancy, we identify and
  study the computation-level redundancy on vision tokens to ensure no
  information loss. Our key insight is that vision tokens from the pretrained
  vision encoder do not necessarily require all the heavy operations (e.g.,
  self-attention, FFNs) in decoder-only LMMs and could be processed more lightly
  with proper designs. We designed a series of experiments to discover and
  progressively squeeze out the vision-related computation redundancy. Based on
  our findings, we propose ProxyV, a novel approach that utilizes proxy vision
  tokens to alleviate the computational burden on original vision tokens. ProxyV
  enhances efficiency without compromising performance and can even yield notable
  performance gains in scenarios with more moderate efficiency improvements.
  Furthermore, the flexibility of ProxyV is demonstrated through its combination
  with token reduction methods to boost efficiency further. The code will be made
  public at this https://github.com/penghao-wu/ProxyV URL.
paper_ai_summary: ProxyV alleviates computational burdens in large multimodal models by using proxy vision tokens, enhancing efficiency without sacrificing performance.
paper_ai_keywords: multimodal models computation-level redundancy vision tokens pretrained vision encoder decoder-only LMMs self-attention FFNs proxy vision tokens ProxyV