INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal
  Large Language Model
With advancements in data availability and computing resources, Multimodal
Large Language Models (MLLMs) have showcased capabilities across various
fields. However, the quadratic complexity of the vision encoder in MLLMs
constrains the resolution of input images. Most current approaches mitigate
this issue by cropping high-resolution images into smaller sub-images, which
are then processed independently by the vision encoder. Despite capturing
sufficient local details, these sub-images lack global context and fail to
interact with one another. To address this limitation, we propose a novel MLLM,
INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA
incorporates two innovative components. First, we introduce a Dual-perspective
Cropping Module (DCM), which ensures that each sub-image contains continuous
details from a local perspective and comprehensive information from a global
perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to
enable the mutual enhancement of global and local features, allowing INF-LLaVA
to effectively process high-resolution images by simultaneously capturing
detailed local information and comprehensive global context. Extensive ablation
studies validate the effectiveness of these components, and experiments on a
diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing
MLLMs. Code and pretrained model are available at
https://github.com/WeihuangLin/INF-LLaVA.
66a05fa14bb7945d6aadb968
Yiwei Ma
admin_assigned
2024-07-24T08:30:53.788Z
664d562df2103a1ce6e8fe2e
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/A479ztZurIZAm0NUwcCvA.jpeg
mayiwei
user
mayiwei
66a05fa14bb7945d6aadb969
Zhibin Wang
admin_assigned
2024-07-24T08:31:00.337Z
662b8980948b908a00c7a4fe
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/tvlJv_9N9_3D9BV23hLHx.png
Zhibin Wang
user
gesmine
66a05fa14bb7945d6aadb96a
Xiaoshuai Sun
66a05fa14bb7945d6aadb96b
Weihuang Lin
admin_assigned
2024-07-24T08:31:14.541Z
6697cccaebf473b4a91b1960
https://cdn-avatars.huggingface.co/v1/production/uploads/6697cccaebf473b4a91b1960/bRdXVMIJvnWpY_JIB5HBm.jpeg
WeihuangLin
user
WeihuangLin
66a05fa14bb7945d6aadb96c
Qiang Zhou
66a05fa14bb7945d6aadb96d
Jiayi Ji
admin_assigned
2024-07-24T08:31:41.083Z
61c7cb3fc4157c1d283da869
/avatars/8707bcb8b4e3725fe43115a88fe14be9.svg
Jiayi Ji
user
amy1254
66a05fa14bb7945d6aadb96e
Rongrong Ji
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal
  Large Language Model
With advancements in data availability and computing resources, Multimodal
Large Language Models (MLLMs) have showcased capabilities across various
fields. However, the quadratic complexity of the vision encoder in MLLMs
constrains the resolution of input images. Most current approaches mitigate
this issue by cropping high-resolution images into smaller sub-images, which
are then processed independently by the vision encoder. Despite capturing
sufficient local details, these sub-images lack global context and fail to
interact with one another. To address this limitation, we propose a novel MLLM,
INF-LLaVA, designed for effective high-resolution image perception. INF-LLaVA
incorporates two innovative components. First, we introduce a Dual-perspective
Cropping Module (DCM), which ensures that each sub-image contains continuous
details from a local perspective and comprehensive information from a global
perspective. Second, we introduce Dual-perspective Enhancement Module (DEM) to
enable the mutual enhancement of global and local features, allowing INF-LLaVA
to effectively process high-resolution images by simultaneously capturing
detailed local information and comprehensive global context. Extensive ablation
studies validate the effectiveness of these components, and experiments on a
diverse set of benchmarks demonstrate that INF-LLaVA outperforms existing
MLLMs. Code and pretrained model are available at
https://github.com/WeihuangLin/INF-LLaVA.
INF-LLaVA, a novel Multimodal Large Language Model, uses a Dual-perspective Cropping Module and Dual-perspective Enhancement Module to effectively process high-resolution images, capturing both local details and global context.
Multimodal Large Language Models MLLMs vision encoder high-resolution image perception Dual-perspective Cropping Module DCM Dual-perspective Enhancement Module DEM global and local features ablation studies benchmarks