title: INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal
    Large Language Model
summary: With advancements in data availability and computing resources, Multimodal
  Large Language Models (MLLMs) have showcased capabilities across various
  fields. However, the quadratic complexity of the vision encoder in MLLMs
  constrains the resolution of input images. Most current approaches mitigate
  this issue by cropping high-resolution images into smaller sub-images, which
  are then processed independently by the vision encoder. Despite capturing
  sufficient local details, these sub-images lack global context and fail to
  interact with one another. To address this limitation, we propose a novel MLLM,
  INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA
  incorporates two innovative components. First, we introduce a Dual-perspective
  Cropping Module (DCM), which ensures that each sub-image contains continuous
  details from a local perspective and comprehensive information from a global
  perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to
  enable the mutual enhancement of global and local features, allowing INF-LLaVA
  to effectively process high-resolution images by simultaneously capturing
  detailed local information and comprehensive global context. Extensive ablation
  studies validate the effectiveness of these components, and experiments on a
  diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing
  MLLMs. Code and pretrained model are available at
  https://github.com/WeihuangLin/INF-LLaVA.
paper_authors._id: 66a05fa14bb7945d6aadb968
paper_authors.name: Yiwei Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:30:53.788Z
paper_authors._id: 664d562df2103a1ce6e8fe2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/A479ztZurIZAm0NUwcCvA.jpeg
paper_authors.fullname: mayiwei
paper_authors.type: user
paper_authors.user: mayiwei
paper_authors._id: 66a05fa14bb7945d6aadb969
paper_authors.name: Zhibin Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:31:00.337Z
paper_authors._id: 662b8980948b908a00c7a4fe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tvlJv_9N9_3D9BV23hLHx.png
paper_authors.fullname: Zhibin Wang
paper_authors.type: user
paper_authors.user: gesmine
paper_authors._id: 66a05fa14bb7945d6aadb96a
paper_authors.name: Xiaoshuai Sun
paper_authors._id: 66a05fa14bb7945d6aadb96b
paper_authors.name: Weihuang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:31:14.541Z
paper_authors._id: 6697cccaebf473b4a91b1960
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6697cccaebf473b4a91b1960/bRdXVMIJvnWpY_JIB5HBm.jpeg
paper_authors.fullname: WeihuangLin
paper_authors.type: user
paper_authors.user: WeihuangLin
paper_authors._id: 66a05fa14bb7945d6aadb96c
paper_authors.name: Qiang Zhou
paper_authors._id: 66a05fa14bb7945d6aadb96d
paper_authors.name: Jiayi Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-24T08:31:41.083Z
paper_authors._id: 61c7cb3fc4157c1d283da869
paper_authors.avatarUrl: /avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
paper_authors.fullname: Jiayi Ji
paper_authors.type: user
paper_authors.user: amy1254
paper_authors._id: 66a05fa14bb7945d6aadb96e
paper_authors.name: Rongrong Ji
paper_title: INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal
    Large Language Model
paper_summary: With advancements in data availability and computing resources, Multimodal
  Large Language Models (MLLMs) have showcased capabilities across various
  fields. However, the quadratic complexity of the vision encoder in MLLMs
  constrains the resolution of input images. Most current approaches mitigate
  this issue by cropping high-resolution images into smaller sub-images, which
  are then processed independently by the vision encoder. Despite capturing
  sufficient local details, these sub-images lack global context and fail to
  interact with one another. To address this limitation, we propose a novel MLLM,
  INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA
  incorporates two innovative components. First, we introduce a Dual-perspective
  Cropping Module (DCM), which ensures that each sub-image contains continuous
  details from a local perspective and comprehensive information from a global
  perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to
  enable the mutual enhancement of global and local features, allowing INF-LLaVA
  to effectively process high-resolution images by simultaneously capturing
  detailed local information and comprehensive global context. Extensive ablation
  studies validate the effectiveness of these components, and experiments on a
  diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing
  MLLMs. Code and pretrained model are available at
  https://github.com/WeihuangLin/INF-LLaVA.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) vision encoder high-resolution images sub-images Dual-perspective Cropping Module (DCM) Dual-perspective Enhancement Module (DEM) global context local details local perspective comprehensive information ablation studies benchmarks