title: Learning to Inference Adaptively for Multimodal Large Language Models
summary: Multimodal Large Language Models (MLLMs) have shown impressive capabilities
  in reasoning, yet come with substantial computational cost, limiting their
  deployment in resource-constrained settings. Despite recent efforts on
  improving the efficiency of MLLMs, prior solutions fall short in responding to
  varying runtime conditions, in particular changing resource availability (e.g.,
  contention due to the execution of other programs on the device). To bridge
  this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
  dynamically reconfigure operations in an MLLM during inference, accounting for
  the input data and a latency budget. We conduct extensive experiments across
  benchmarks involving question-answering, reasoning, and hallucination. Our
  results show that AdaLLaVA effectively adheres to input latency budget,
  achieving varying accuracy and latency tradeoffs at runtime. Further, we
  demonstrate that AdaLLaVA adapts to both input latency and content, can be
  integrated with token selection for enhanced efficiency, and generalizes across
  MLLMs. Our project webpage with code release is at
  https://zhuoyan-xu.github.io/ada-llava/.
paper_authors._id: 67daf18da3b9c06eb0727795
paper_authors.name: Zhuoyan Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-20T10:45:58.688Z
paper_authors._id: 64be0648b567ae97c3493af6
paper_authors.avatarUrl: /avatars/c932a4d49e2c9d72898d190de6545f92.svg
paper_authors.fullname: Zhuoyan Xu
paper_authors.type: user
paper_authors.user: zhuoyanxu
paper_authors._id: 67daf18da3b9c06eb0727796
paper_authors.name: Khoi Duc Nguyen
paper_authors._id: 67daf18da3b9c06eb0727797
paper_authors.name: Preeti Mukherjee
paper_authors._id: 67daf18da3b9c06eb0727798
paper_authors.name: Saurabh Bagchi
paper_authors._id: 67daf18da3b9c06eb0727799
paper_authors.name: Somali Chaterji
paper_authors._id: 67daf18da3b9c06eb072779a
paper_authors.name: Yingyu Liang
paper_authors._id: 67daf18da3b9c06eb072779b
paper_authors.name: Yin Li
paper_title: Learning to Inference Adaptively for Multimodal Large Language Models
paper_summary: Multimodal Large Language Models (MLLMs) have shown impressive capabilities
  in reasoning, yet come with substantial computational cost, limiting their
  deployment in resource-constrained settings. Despite recent efforts on
  improving the efficiency of MLLMs, prior solutions fall short in responding to
  varying runtime conditions, in particular changing resource availability (e.g.,
  contention due to the execution of other programs on the device). To bridge
  this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
  dynamically reconfigure operations in an MLLM during inference, accounting for
  the input data and a latency budget. We conduct extensive experiments across
  benchmarks involving question-answering, reasoning, and hallucination. Our
  results show that AdaLLaVA effectively adheres to input latency budget,
  achieving varying accuracy and latency tradeoffs at runtime. Further, we
  demonstrate that AdaLLaVA adapts to both input latency and content, can be
  integrated with token selection for enhanced efficiency, and generalizes across
  MLLMs. Our project webpage with code release is at
  https://zhuoyan-xu.github.io/ada-llava/.
paper_ai_summary: An adaptive inference framework, AdaLLaVA, dynamically reconfigures multimodal large language models during inference to optimize for both accuracy and latency under varying runtime conditions.
paper_ai_keywords: Multimodal Large Language Models adaptive inference dynamic reconfiguration input latency budget latency tradeoffs token selection