title: On Speculative Decoding for Multimodal Large Language Models
summary: Inference with Multimodal Large Language Models (MLLMs) is slow due to their
  large-language-model backbone which suffers from memory bandwidth bottleneck
  and generates tokens auto-regressively. In this paper, we explore the
  application of speculative decoding to enhance the inference efficiency of
  MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
  serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
  the need for image tokens and their associated processing components from the
  draft model. Our experiments across three different tasks show that speculative
  decoding can achieve a memory-bound speedup of up to 2.37times using a 115M
  parameter language model that we trained from scratch. Additionally, we
  introduce a compact LLaVA draft model incorporating an image adapter, which
  shows marginal performance gains in image captioning while maintaining
  comparable results in other tasks.
paper_authors._id: 661dfac00d0c77a84194522d
paper_authors.name: Mukul Gagrani
paper_authors._id: 661dfac00d0c77a84194522e
paper_authors.name: Raghavv Goel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-18T19:35:08.743Z
paper_authors._id: 649b6eb2f7cc759ab756adaf
paper_authors.avatarUrl: /avatars/a0f7c7345dd653887122200fbe375c2b.svg
paper_authors.fullname: Raghavv Goel
paper_authors.type: user
paper_authors.user: RaghavvGoel
paper_authors._id: 661dfac00d0c77a84194522f
paper_authors.name: Wonseok Jeon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T11:04:51.108Z
paper_authors._id: 64b8096b08e2452d18ef1a28
paper_authors.avatarUrl: /avatars/6ccb615abe479c33872e44a8c8792db1.svg
paper_authors.fullname: Wonseok Jeon
paper_authors.type: user
paper_authors.user: jeonwonseok0125
paper_authors._id: 661dfac00d0c77a841945230
paper_authors.name: Junyoung Park
paper_authors._id: 661dfac00d0c77a841945231
paper_authors.name: Mingu Lee
paper_authors._id: 661dfac00d0c77a841945232
paper_authors.name: Christopher Lott
paper_title: On Speculative Decoding for Multimodal Large Language Models
paper_summary: Inference with Multimodal Large Language Models (MLLMs) is slow due to their
  large-language-model backbone which suffers from memory bandwidth bottleneck
  and generates tokens auto-regressively. In this paper, we explore the
  application of speculative decoding to enhance the inference efficiency of
  MLLMs, specifically the LLaVA 7B model. We show that a language-only model can
  serve as a good draft model for speculative decoding with LLaVA 7B, bypassing
  the need for image tokens and their associated processing components from the
  draft model. Our experiments across three different tasks show that speculative
  decoding can achieve a memory-bound speedup of up to 2.37times using a 115M
  parameter language model that we trained from scratch. Additionally, we
  introduce a compact LLaVA draft model incorporating an image adapter, which
  shows marginal performance gains in image captioning while maintaining
  comparable results in other tasks.
paper_ai_summary: Speculative decoding enhances the inference efficiency of Multimodal Large Language Models (MLLMs), specifically LLaVA 7B, by using a language-only draft model and a compact LLaVA model with an image adapter.
paper_ai_keywords: Multimodal Large Language Models MLLMs speculative decoding LLaVA 7B draft model image adapter image captioning memory-bound speedup