5

Figure 2: Detailed image descriptions

by training the model with more high-quality, aligned image-text pairs, or aligning with more
advanced LLMs in the future.
Inadequate perception capacities. MiniGPT-4’s visual perception remains limited. It may struggle
to recognize detailed textual information from images, and differentiate spatial localization. This
limitation may stem from several factors: 1) A lack of sufﬁcient aligned image-text data containing
adequate information such as spatial localization and optical character annotations. This issue could
be alleviated by training on more well-aligned and rich data. 2) The frozen Q-former used in the visual
encoder may lose some essential features, such as visual-spatial grounding. This could potentially be
improved by replacing it with a stronger visual perception model. 3) Training only one projection
layer might not provide enough capacity to learn extensive visual-text alignment.

6
MiniGPT-4- Enhancing Vision-Language Understanding with Advanced Large Language Models