3.3 Second-stage ﬁnetuning

During the second stage, we ﬁnetune our pretrained model with the curated high-quality image-text
pairs. During the ﬁnetuning, we use the predeﬁned prompts in the following template:
###Human: <Img><ImageFeature></Img> <Instruction> ###Assistant:
In this prompt, <Instruction> represents a randomly sampled instruction from our predeﬁned instruc-
tion set containing variant forms of instructions such as “Describe this image in detail” or “Could
you describe the contents of this image for me”. It is important to note that we do not calculate the
regression loss for this speciﬁc text-image prompt.
As a result, MiniGPT-4 is now capable of producing more natural and reliable responses. Furthermore,
we have observed that the model’s ﬁne-tuning process is remarkably efﬁcient, only requiring a mere
400 training steps with batch size of 12, which takes a brief 7 minutes to complete with a single A100
GPU.

4 Demonstrations:
MiniGPT-4- Enhancing Vision-Language Understanding with Advanced Large Language Models