title: TinyLLaVA: A Framework of Small-scale Large Multimodal Models
summary: We present the TinyLLaVA framework that provides a unified perspective in
  designing and analyzing the small-scale Large Multimodal Models (LMMs). We
  empirically study the effects of different vision encoders, connection modules,
  language models, training data and training recipes. Our extensive experiments
  showed that better quality of data combined with better training recipes,
  smaller LMMs can consistently achieve on-par performances compared to bigger
  LMMs. Under our framework, we train a family of small-scale LMMs. Our best
  model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B
  models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as
  baselines for future research in terms of data scaling, training setups and
  model selections. Our model weights and codes will be made public.
paper_authors._id: 65d810553381a5be2779b3d3
paper_authors.name: Baichuan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-23T12:25:13.725Z
paper_authors._id: 646325085897b675c65aea0f
paper_authors.avatarUrl: /avatars/28ce7388f9318b49bdd0a5594c0f6732.svg
paper_authors.fullname: Baichuan Zhou
paper_authors.type: user
paper_authors.user: bczhou
paper_authors._id: 65d810553381a5be2779b3d4
paper_authors.name: Ying Hu
paper_authors._id: 65d810553381a5be2779b3d5
paper_authors.name: Xi Weng
paper_authors._id: 65d810553381a5be2779b3d6
paper_authors.name: Junlong Jia
paper_authors._id: 65d810553381a5be2779b3d7
paper_authors.name: Jie Luo
paper_authors._id: 65d810553381a5be2779b3d8
paper_authors.name: Xien Liu
paper_authors._id: 65d810553381a5be2779b3d9
paper_authors.name: Ji Wu
paper_authors._id: 65d810553381a5be2779b3da
paper_authors.name: Lei Huang
paper_title: TinyLLaVA: A Framework of Small-scale Large Multimodal Models
paper_summary: We present the TinyLLaVA framework that provides a unified perspective in
  designing and analyzing the small-scale Large Multimodal Models (LMMs). We
  empirically study the effects of different vision encoders, connection modules,
  language models, training data and training recipes. Our extensive experiments
  showed that better quality of data combined with better training recipes,
  smaller LMMs can consistently achieve on-par performances compared to bigger
  LMMs. Under our framework, we train a family of small-scale LMMs. Our best
  model, TinyLLaVA-3.1B, achieves better overall performance against existing 7B
  models such as LLaVA-1.5 and Qwen-VL. We hope our findings can serve as
  baselines for future research in terms of data scaling, training setups and
  model selections. Our model weights and codes will be made public.
paper_ai_keywords: TinyLLaVA framework Large Multimodal Models (LMMs) vision encoders connection modules language models training data training recipes TinyLLaVA-3.1B LLaVA-1.5 Qwen-VL