title: LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
    Language Model
summary: We train a suite of multimodal foundation models (MMFM) using the popular
  LLaVA framework with the recently released Gemma family of large language
  models (LLMs). Of particular interest is the 2B parameter Gemma model, which
  provides opportunities to construct capable small-scale MMFMs. In line with
  findings from other papers in this space, we test the effect of ablating three
  design features: pretraining the connector, utilizing a more powerful image
  backbone, and increasing the size of the language backbone. The resulting
  models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
  evaluations, but fail to improve past the current comparably sized SOTA models.
  Closer analysis of performance shows mixed effects; skipping pretraining tends
  to reduce performance, larger vision models sometimes improve performance, and
  increasing language model size has inconsistent effects. We publicly release
  training recipes, code and weights for our models for the LLaVA-Gemma models.
paper_authors._id: 660cbcd9852e9d33f1810e1a
paper_authors.name: Musashi Hinck
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-04T07:50:02.643Z
paper_authors._id: 65de48ac04b9352580c257fb
paper_authors.avatarUrl: /avatars/2cfa15570cf5c47473a0bd1e124886bf.svg
paper_authors.fullname: Musashi Hinck
paper_authors.type: user
paper_authors.user: musashihinck
paper_authors._id: 660cbcd9852e9d33f1810e1b
paper_authors.name: Matthew L. Olson
paper_authors._id: 660cbcd9852e9d33f1810e1c
paper_authors.name: David Cobbley
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-04T07:50:14.811Z
paper_authors._id: 6515c52426224cf8b1e9f5ca
paper_authors.avatarUrl: /avatars/38eeac1e8a4dd37152c556af60814c65.svg
paper_authors.fullname: Dave Cobbley
paper_authors.type: user
paper_authors.user: djcobble
paper_authors._id: 660cbcd9852e9d33f1810e1d
paper_authors.name: Shao-Yen Tseng
paper_authors._id: 660cbcd9852e9d33f1810e1e
paper_authors.name: Vasudev Lal
paper_title: LLaVA-Gemma: Accelerating Multimodal Foundation Models with a Compact
    Language Model
paper_summary: We train a suite of multimodal foundation models (MMFM) using the popular
  LLaVA framework with the recently released Gemma family of large language
  models (LLMs). Of particular interest is the 2B parameter Gemma model, which
  provides opportunities to construct capable small-scale MMFMs. In line with
  findings from other papers in this space, we test the effect of ablating three
  design features: pretraining the connector, utilizing a more powerful image
  backbone, and increasing the size of the language backbone. The resulting
  models, which we call LLaVA-Gemma, exhibit moderate performance on an array of
  evaluations, but fail to improve past the current comparably sized SOTA models.
  Closer analysis of performance shows mixed effects; skipping pretraining tends
  to reduce performance, larger vision models sometimes improve performance, and
  increasing language model size has inconsistent effects. We publicly release
  training recipes, code and weights for our models for the LLaVA-Gemma models.
paper_ai_keywords: multimodal foundation models (MMFM) LLaVA framework Gemma family of large language models (LLMs) 2B parameter Gemma model LLaVA-Gemma pretraining the connector image backbone language backbone