title: Zero-Shot Vision Encoder Grafting via LLM Surrogates
summary: Vision language models (VLMs) typically pair a modestly sized vision encoder
  with a large language model (LLM), e.g., Llama-70B, making the decoder the
  primary computational burden during training. To reduce costs, a potential
  promising strategy is to first train the vision encoder using a small language
  model before transferring it to the large one. We construct small "surrogate
  models" that share the same embedding space and representation language as the
  large target LLM by directly inheriting its shallow layers. Vision encoders
  trained on the surrogate can then be directly transferred to the larger model,
  a process we call zero-shot grafting -- when plugged directly into the
  full-size target LLM, the grafted pair surpasses the encoder-surrogate pair
  and, on some benchmarks, even performs on par with full decoder training with
  the target LLM. Furthermore, our surrogate training approach reduces overall
  VLM training costs by ~45% when using Llama-70B as the decoder.
paper_authors._id: 6837befa0aa18c6f96f4a9a5
paper_authors.name: Kaiyu Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:50:48.513Z
paper_authors._id: 640d0dbc8036cc2142273a83
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640d0dbc8036cc2142273a83/TQHyWWMGuXJKEM6fTPf-z.png
paper_authors.fullname: Kaiyu Yue
paper_authors.type: user
paper_authors.user: kaiyuyue
paper_authors._id: 6837befa0aa18c6f96f4a9a6
paper_authors.name: Vasu Singla
paper_authors._id: 6837befa0aa18c6f96f4a9a7
paper_authors.name: Menglin Jia
paper_authors._id: 6837befa0aa18c6f96f4a9a8
paper_authors.name: John Kirchenbauer
paper_authors._id: 6837befa0aa18c6f96f4a9a9
paper_authors.name: Rifaa Qadri
paper_authors._id: 6837befa0aa18c6f96f4a9aa
paper_authors.name: Zikui Cai
paper_authors._id: 6837befa0aa18c6f96f4a9ab
paper_authors.name: Abhinav Bhatele
paper_authors._id: 6837befa0aa18c6f96f4a9ac
paper_authors.name: Furong Huang
paper_authors._id: 6837befa0aa18c6f96f4a9ad
paper_authors.name: Tom Goldstein
paper_title: Zero-Shot Vision Encoder Grafting via LLM Surrogates
paper_summary: Vision language models (VLMs) typically pair a modestly sized vision encoder
  with a large language model (LLM), e.g., Llama-70B, making the decoder the
  primary computational burden during training. To reduce costs, a potential
  promising strategy is to first train the vision encoder using a small language
  model before transferring it to the large one. We construct small "surrogate
  models" that share the same embedding space and representation language as the
  large target LLM by directly inheriting its shallow layers. Vision encoders
  trained on the surrogate can then be directly transferred to the larger model,
  a process we call zero-shot grafting -- when plugged directly into the
  full-size target LLM, the grafted pair surpasses the encoder-surrogate pair
  and, on some benchmarks, even performs on par with full decoder training with
  the target LLM. Furthermore, our surrogate training approach reduces overall
  VLM training costs by ~45% when using Llama-70B as the decoder.
paper_ai_summary: The approach of training vision encoders with small surrogate models before transferring them to large language models reduces training costs and enhances performance.
paper_ai_keywords: vision language models vision encoder large language model small language model surrogate models embedding space representation language zero-shot grafting