whose formulation is in Fig. 5. During training, LLM is
required to predict the “task-specific responses” accord-
ing to the “USER-INPUTs” which include “Task-specific
Prompt” and two additional optional conditions like image
and text condition. To avoid over-fitting, we only train stan-
dard LoRA [72] for the LLM with the associable instruction
data. See more details in Appendix.
Let’sThinkOutsidetheBox