images, providing solutions to problems shown in images, teaching users how to
cook based on food photos, etc. In our experiment, we found that only performing
the pretraining on raw image-text pairs could produce unnatural language outputs
that lack coherency including repetition and fragmented sentences. To address
this problem, we curate a high-quality, well-aligned dataset in the second stage to
ﬁnetune our model using a conversational template. This step proved crucial for
augmenting the model’s generation reliability and overall usability. Notably, our
model is highly computationally efﬁcient, as we only train a projection layer utiliz-
ing approximately 5 million aligned image-text pairs. Our code, pre-trained model,
and collected dataset are available at https://minigpt-4.github.io/.
MiniGPT-4- Enhancing Vision-Language Understanding with Advanced Large Language Models