InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model
  Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant
advancements, yet its progression has been hindered by challenges in
comprehending fine-grained visual content due to limited resolution. Recent
efforts have aimed to enhance the high-resolution understanding capabilities of
LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and
constrained to a relatively narrow resolution range. This paper represents
InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM
resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently,
considering the ultra-high resolution may not be necessary in all scenarios, it
supports a wide range of diverse resolutions from 336 pixels to 4K standard,
significantly broadening its scope of applicability. Specifically, this
research advances the patch division paradigm by introducing a novel extension:
dynamic resolution with automatic patch configuration. It maintains the
training image aspect ratios while automatically varying patch counts and
configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x
336), leading to dynamic training resolution from 336 pixels to 4K standard.
Our research demonstrates that scaling training resolution up to 4K HD leads to
consistent performance enhancements without hitting the ceiling of potential
improvements. InternLM-XComposer2-4KHD shows superb capability that matches or
even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The
InternLM-XComposer2-4KHD model series with 7B parameters are publicly available
at https://github.com/InternLM/InternLM-XComposer.
6615f0031ddd756333b2f134
Xiaoyi Dong
6615f0031ddd756333b2f135
Pan Zhang
6615f0031ddd756333b2f136
Yuhang Zang
admin_assigned
2024-04-22T14:09:41.451Z
63859cf3b2906edaf83af9f0
https://cdn-avatars.huggingface.co/v1/production/uploads/63859cf3b2906edaf83af9f0/mCgynEtoXdILvzyoPexii.png
Yuhang Zang
user
yuhangzang
6615f0031ddd756333b2f137
Yuhang Cao
admin_assigned
2024-04-22T14:09:51.134Z
65000bef18830fabea469fdd
/avatars/b320c77dfad039d9f9c54127f610d44f.svg
Cao Yuhang
user
yhcao
6615f0031ddd756333b2f138
Bin Wang
admin_assigned
2024-04-22T14:09:57.885Z
5e49e8cf37cb5b49818287ae
https://cdn-avatars.huggingface.co/v1/production/uploads/5e49e8cf37cb5b49818287ae/IV9b5Z70NhgmBNfAlc_co.jpeg
Bin Wang
user
binwang
6615f0031ddd756333b2f139
Linke Ouyang
6615f0031ddd756333b2f13a
Songyang Zhang
claimed_verified
2024-08-05T08:00:27.592Z
630716d11801ecc7d2595021
/avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
Songyang Zhang
user
zsytony
6615f0031ddd756333b2f13b
Haodong Duan
admin_assigned
2024-04-22T14:10:39.088Z
63ee1379190ddd6214efd73a
https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
HAODONG DUAN
user
KennyUTC
6615f0031ddd756333b2f13c
Wenwei Zhang
admin_assigned
2024-04-22T14:10:46.298Z
64e8505321540e1da3226b54
/avatars/18958b8406d1ce492b54c1c839f18c54.svg
Wenwei Zhang
user
ZwwWayne
6615f0031ddd756333b2f13d
Yining Li
claimed_verified
2025-04-22T10:17:56.755Z
649988726677f66c2b486392
/avatars/b649a77370660e129726e29504daba34.svg
Yining Li
user
ly015
6615f0031ddd756333b2f13e
Hang Yan
6615f0031ddd756333b2f13f
Yang Gao
6615f0031ddd756333b2f140
Zhe Chen
claimed_verified
2024-12-09T14:49:47.535Z
64119264f0f81eb569e0d569
/avatars/00dccedea51ac7d2f398ca9d65a8e78b.svg
Zhe Chen
user
czczup
6615f0031ddd756333b2f141
Xinyue Zhang
6615f0031ddd756333b2f142
Wei Li
6615f0031ddd756333b2f143
Jingwen Li
admin_assigned
2024-04-22T14:11:56.832Z
64da1d1e7f8116a6ab1f5181
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/blOlCW3k-GR4zV9zCA94s.jpeg
Jingwen Li
user
mersso
6615f0031ddd756333b2f144
Wenhai Wang
6615f0031ddd756333b2f145
Kai Chen
6615f0031ddd756333b2f146
Conghui He
6615f0031ddd756333b2f147
Xingcheng Zhang
6615f0031ddd756333b2f148
Jifeng Dai
6615f0031ddd756333b2f149
Yu Qiao
6615f0031ddd756333b2f14a
Dahua Lin
admin_assigned
2024-04-22T14:12:13.665Z
636317ed80c1a705a6eff396
/avatars/3db090e101b916d9256d0d3e043db71d.svg
Dahua Lin
user
lindahua
6615f0031ddd756333b2f14b
Jiaqi Wang
claimed_verified
2024-06-09T20:06:20.449Z
64b4eec4faa3181a5eab9c46
/avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
Jiaqi Wang
user
myownskyW7
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model
  Handling Resolutions from 336 Pixels to 4K HD
The Large Vision-Language Model (LVLM) field has seen significant
advancements, yet its progression has been hindered by challenges in
comprehending fine-grained visual content due to limited resolution. Recent
efforts have aimed to enhance the high-resolution understanding capabilities of
LVLMs, yet they remain capped at approximately 1500 x 1500 pixels and
constrained to a relatively narrow resolution range. This paper represents
InternLM-XComposer2-4KHD, a groundbreaking exploration into elevating LVLM
resolution capabilities up to 4K HD (3840 x 1600) and beyond. Concurrently,
considering the ultra-high resolution may not be necessary in all scenarios, it
supports a wide range of diverse resolutions from 336 pixels to 4K standard,
significantly broadening its scope of applicability. Specifically, this
research advances the patch division paradigm by introducing a novel extension:
dynamic resolution with automatic patch configuration. It maintains the
training image aspect ratios while automatically varying patch counts and
configuring layouts based on a pre-trained Vision Transformer (ViT) (336 x
336), leading to dynamic training resolution from 336 pixels to 4K standard.
Our research demonstrates that scaling training resolution up to 4K HD leads to
consistent performance enhancements without hitting the ceiling of potential
improvements. InternLM-XComposer2-4KHD shows superb capability that matches or
even surpasses GPT-4V and Gemini Pro in 10 of the 16 benchmarks. The
InternLM-XComposer2-4KHD model series with 7B parameters are publicly available
at https://github.com/InternLM/InternLM-XComposer.
InternLM-XComposer2-4KHD enhances high-resolution understanding capabilities in LVLMs up to 4K, surpassing existing models with dynamic patch configuration and consistent performance improvements.
Large Vision-Language Model LVLM high-resolution understanding patch division paradigm dynamic resolution automatic patch configuration Vision Transformer ViT training image aspect ratios consistent performance enhancements GPT-4V Gemini Pro