title: LLaVA-φ: Efficient Multi-Modal Assistant with Small Language Model
summary: In this paper, we introduce LLaVA-phi (LLaVA-Phi), an efficient
  multi-modal assistant that harnesses the power of the recently advanced small
  language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a
  notable advancement in the realm of compact multi-modal models. It demonstrates
  that even smaller language models, with as few as 2.7B parameters, can
  effectively engage in intricate dialogues that integrate both textual and
  visual elements, provided they are trained with high-quality corpora. Our model
  delivers commendable performance on publicly available benchmarks that
  encompass visual comprehension, reasoning, and knowledge-based perception.
  Beyond its remarkable performance in multi-modal dialogue tasks, our model
  opens new avenues for applications in time-sensitive environments and systems
  that require real-time interaction, such as embodied agents. It highlights the
  potential of smaller language models to achieve sophisticated levels of
  understanding and interaction, while maintaining greater resource
  efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
paper_authors._id: 65978c204dbb962b7c5126ba
paper_authors.name: Yichen Zhu
paper_authors._id: 65978c204dbb962b7c5126bb
paper_authors.name: Minjie Zhu
paper_authors._id: 65978c204dbb962b7c5126bc
paper_authors.name: Ning Liu
paper_authors._id: 65978c204dbb962b7c5126bd
paper_authors.name: Zhicai Ou
paper_authors._id: 65978c204dbb962b7c5126be
paper_authors.name: Xiaofeng Mou
paper_authors._id: 65978c204dbb962b7c5126bf
paper_authors.name: Jian Tang
paper_title: LLaVA-φ: Efficient Multi-Modal Assistant with Small Language Model
paper_summary: In this paper, we introduce LLaVA-phi (LLaVA-Phi), an efficient
  multi-modal assistant that harnesses the power of the recently advanced small
  language model, Phi-2, to facilitate multi-modal dialogues. LLaVA-Phi marks a
  notable advancement in the realm of compact multi-modal models. It demonstrates
  that even smaller language models, with as few as 2.7B parameters, can
  effectively engage in intricate dialogues that integrate both textual and
  visual elements, provided they are trained with high-quality corpora. Our model
  delivers commendable performance on publicly available benchmarks that
  encompass visual comprehension, reasoning, and knowledge-based perception.
  Beyond its remarkable performance in multi-modal dialogue tasks, our model
  opens new avenues for applications in time-sensitive environments and systems
  that require real-time interaction, such as embodied agents. It highlights the
  potential of smaller language models to achieve sophisticated levels of
  understanding and interaction, while maintaining greater resource
  efficiency.The project is available at {https://github.com/zhuyiche/llava-phi}.
paper_ai_summary: A compact multi-modal assistant, LLaVA-Phi, using a smaller language model, Phi-2, achieves high performance in visual and multi-modal dialog tasks with resource efficiency.
paper_ai_keywords: multi-modal assistant Phi-2 multi-modal dialogues visual comprehension reasoning knowledge-based perception embodied agents