title: Parameter-Inverted Image Pyramid Networks for Visual Perception and
    Multimodal Understanding
summary: Image pyramids are widely adopted in top-performing methods to obtain
  multi-scale features for precise visual perception and understanding. However,
  current image pyramids use the same large-scale model to process multiple
  resolutions of images, leading to significant computational cost. To address
  this challenge, we propose a novel network architecture, called
  Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses
  pretrained models (ViTs or CNNs) as branches to process multi-scale images,
  where images of higher resolutions are processed by smaller network branches to
  balance computational cost and performance. To integrate information from
  different spatial scales, we further propose a novel cross-branch feature
  interaction mechanism. To validate PIIP, we apply it to various perception
  models and a representative multimodal large language model called LLaVA, and
  conduct extensive experiments on various tasks such as object detection,
  segmentation, image classification and multimodal understanding. PIIP achieves
  superior performance compared to single-branch and existing multi-resolution
  approaches with lower computational cost. When applied to InternViT-6B, a
  large-scale vision foundation model, PIIP can improve its performance by 1%-2%
  on detection and segmentation with only 40%-60% of the original computation,
  finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For
  multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and
  74.5% on MMBench with only 2.8M training data. Our code is released at
  https://github.com/OpenGVLab/PIIP.
paper_authors._id: 6788c7156bc665d65caf8b3c
paper_authors.name: Zhaokai Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-16T08:49:11.408Z
paper_authors._id: 665d4b515fdfe8f923e347a7
paper_authors.avatarUrl: /avatars/d114b24c02dadfca0a8aee104755a8ec.svg
paper_authors.fullname: Zhaokai Wang
paper_authors.type: user
paper_authors.user: wzk1015
paper_authors._id: 6788c7156bc665d65caf8b3d
paper_authors.name: Xizhou Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:51:26.043Z
paper_authors._id: 64ae2359179421d320b1694b
paper_authors.avatarUrl: /avatars/c387a75191005bcaa473091de5383a10.svg
paper_authors.fullname: Xizhou Zhu
paper_authors.type: user
paper_authors.user: Einsiedler
paper_authors._id: 6788c7156bc665d65caf8b3e
paper_authors.name: Xue Yang
paper_authors._id: 6788c7156bc665d65caf8b3f
paper_authors.name: Gen Luo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:51:36.685Z
paper_authors._id: 650aac7c23196fb2d86a0b37
paper_authors.avatarUrl: /avatars/418035a2e8f514118bc67d16ee41b6b0.svg
paper_authors.fullname: Gen Luo
paper_authors.type: user
paper_authors.user: favor123
paper_authors._id: 6788c7156bc665d65caf8b40
paper_authors.name: Hao Li
paper_authors._id: 6788c7156bc665d65caf8b41
paper_authors.name: Changyao Tian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:51:46.333Z
paper_authors._id: 64b7475efa7eabaae5f7ba94
paper_authors.avatarUrl: /avatars/346e53b345ccd9e8557ab8d2ec17a8f3.svg
paper_authors.fullname: Changyao Tian
paper_authors.type: user
paper_authors.user: Changyao
paper_authors._id: 6788c7156bc665d65caf8b42
paper_authors.name: Wenhan Dou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:51:57.729Z
paper_authors._id: 66efe658de163a536aa84178
paper_authors.avatarUrl: /avatars/fddc42450cabf41ca1ab2f70b185f51c.svg
paper_authors.fullname: dou wenhan
paper_authors.type: user
paper_authors.user: douwh
paper_authors._id: 6788c7156bc665d65caf8b43
paper_authors.name: Junqi Ge
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:52:04.125Z
paper_authors._id: 6695ee745e5e72a434fdfbc2
paper_authors.avatarUrl: /avatars/01c0af2dac291ed4f52615e07094ea93.svg
paper_authors.fullname: Junqi Ge
paper_authors.type: user
paper_authors.user: gejq16148
paper_authors._id: 6788c7156bc665d65caf8b44
paper_authors.name: Lewei Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:52:10.641Z
paper_authors._id: 65ead3ea908526a39082e641
paper_authors.avatarUrl: /avatars/dcf870695fd56b06ca03d82f831e9019.svg
paper_authors.fullname: Lewei Lu
paper_authors.type: user
paper_authors.user: luotto
paper_authors._id: 6788c7156bc665d65caf8b45
paper_authors.name: Yu Qiao
paper_authors._id: 6788c7156bc665d65caf8b46
paper_authors.name: Jifeng Dai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-16T08:52:18.008Z
paper_authors._id: 64686f7172d9180d4ac8b4e4
paper_authors.avatarUrl: /avatars/db67dd6c4b2b41054ddcce5a18ade6f8.svg
paper_authors.fullname: Jifeng Dai
paper_authors.type: user
paper_authors.user: daijifeng
paper_title: Parameter-Inverted Image Pyramid Networks for Visual Perception and
    Multimodal Understanding
paper_summary: Image pyramids are widely adopted in top-performing methods to obtain
  multi-scale features for precise visual perception and understanding. However,
  current image pyramids use the same large-scale model to process multiple
  resolutions of images, leading to significant computational cost. To address
  this challenge, we propose a novel network architecture, called
  Parameter-Inverted Image Pyramid Networks (PIIP). Specifically, PIIP uses
  pretrained models (ViTs or CNNs) as branches to process multi-scale images,
  where images of higher resolutions are processed by smaller network branches to
  balance computational cost and performance. To integrate information from
  different spatial scales, we further propose a novel cross-branch feature
  interaction mechanism. To validate PIIP, we apply it to various perception
  models and a representative multimodal large language model called LLaVA, and
  conduct extensive experiments on various tasks such as object detection,
  segmentation, image classification and multimodal understanding. PIIP achieves
  superior performance compared to single-branch and existing multi-resolution
  approaches with lower computational cost. When applied to InternViT-6B, a
  large-scale vision foundation model, PIIP can improve its performance by 1%-2%
  on detection and segmentation with only 40%-60% of the original computation,
  finally achieving 60.0 box AP on MS COCO and 59.7 mIoU on ADE20K. For
  multimodal understanding, our PIIP-LLaVA achieves 73.0% accuracy on TextVQA and
  74.5% on MMBench with only 2.8M training data. Our code is released at
  https://github.com/OpenGVLab/PIIP.
paper_ai_keywords: Parameter-Inverted Image Pyramid Networks (PIIP) pretrained models ViTs (Vision Transformers) CNNs (Convolutional Neural Networks) cross-branch feature interaction mechanism object detection segmentation image classification multimodal understanding LLaVA InternViT-6B MS COCO ADE20K TextVQA MMBench