Learning to Inference Adaptively for Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have shown impressive capabilities
in reasoning, yet come with substantial computational cost, limiting their
deployment in resource-constrained settings. Despite recent efforts on
improving the efficiency of MLLMs, prior solutions fall short in responding to
varying runtime conditions, in particular changing resource availability (e.g.,
contention due to the execution of other programs on the device). To bridge
this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
dynamically reconfigure operations in an MLLM during inference, accounting for
the input data and a latency budget. We conduct extensive experiments across
benchmarks involving question-answering, reasoning, and hallucination. Our
results show that AdaLLaVA effectively adheres to input latency budget,
achieving varying accuracy and latency tradeoffs at runtime. Further, we
demonstrate that AdaLLaVA adapts to both input latency and content, can be
integrated with token selection for enhanced efficiency, and generalizes across
MLLMs. Our project webpage with code release is at
https://zhuoyan-xu.github.io/ada-llava/.
67daf18da3b9c06eb0727795
Zhuoyan Xu
claimed_verified
2025-03-20T10:45:58.688Z
64be0648b567ae97c3493af6
/avatars/c932a4d49e2c9d72898d190de6545f92.svg
Zhuoyan Xu
user
zhuoyanxu
67daf18da3b9c06eb0727796
Khoi Duc Nguyen
67daf18da3b9c06eb0727797
Preeti Mukherjee
67daf18da3b9c06eb0727798
Saurabh Bagchi
67daf18da3b9c06eb0727799
Somali Chaterji
67daf18da3b9c06eb072779a
Yingyu Liang
67daf18da3b9c06eb072779b
Yin Li
Learning to Inference Adaptively for Multimodal Large Language Models
Multimodal Large Language Models (MLLMs) have shown impressive capabilities
in reasoning, yet come with substantial computational cost, limiting their
deployment in resource-constrained settings. Despite recent efforts on
improving the efficiency of MLLMs, prior solutions fall short in responding to
varying runtime conditions, in particular changing resource availability (e.g.,
contention due to the execution of other programs on the device). To bridge
this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
dynamically reconfigure operations in an MLLM during inference, accounting for
the input data and a latency budget. We conduct extensive experiments across
benchmarks involving question-answering, reasoning, and hallucination. Our
results show that AdaLLaVA effectively adheres to input latency budget,
achieving varying accuracy and latency tradeoffs at runtime. Further, we
demonstrate that AdaLLaVA adapts to both input latency and content, can be
integrated with token selection for enhanced efficiency, and generalizes across
MLLMs. Our project webpage with code release is at
https://zhuoyan-xu.github.io/ada-llava/.
An adaptive inference framework, AdaLLaVA, dynamically reconfigures multimodal large language models during inference to optimize for both accuracy and latency under varying runtime conditions.
Multimodal Large Language Models adaptive inference dynamic reconfiguration input latency budget latency tradeoffs token selection