Feast Your Eyes:  Mixture-of-Resolution Adaptation for Multimodal Large Language Models
Gen Luo Yiyi Zhou Yuxin Zhang Xiawu Zheng Xiaoshuai Sun Rongrong Ji
In existing multimodal large language models (MLLMs), image resolution plays a significant role for  granular visual recognition.  However, directly increasing image resolution leads to expensive computational cost for MLLMs.  In this paper, we  reveal that a combination of low- and high-resolution visual features can efficiently mitigate this shortcoming.  Based on this principle, we propose a novel and efficient method for MLLMs, termed Mixture-of-Resolution Adaptation (MRA). In particular, MRA adopts two visual pathways for  images of different resolutions, where  high-resolution visual information is embedded into the low-resolution pathway via the novel mixture-of-resolution adapters (MR-Adapters). This design also   greatly  reduces the input sequence length of MLLMs. To validate MRA, we apply it to a recent MLLM called LLaVA, and term the new model LLaVA-HR. We conduct extensive  experiments on 17 vision-language (VL) tasks, which show that LLaVA-HR outperforms existing MLLMs on 15 VL tasks, e.g., +5.2\% on TextVQA.  More importantly,    both training and inference  of LLaVA-HR remain efficient with MRA, e.g.,  20 training hours and  faster inference speed  than LLaVA-NeXT.  Source codes are  released at: https://github.com/luogen1996/LLaVA-HR.
high-resolution adaptation, multimodal large language models