title: DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for
    Efficient Robot Execution
summary: MLLMs have demonstrated remarkable comprehension and reasoning capabilities
  with complex language and visual data. These advances have spurred the vision
  of establishing a generalist robotic MLLM proficient in understanding complex
  human instructions and accomplishing various embodied tasks. However,
  developing MLLMs for real-world robots is challenging due to the typically
  limited computation and memory capacities available on robotic platforms. In
  contrast, the inference of MLLMs involves storing billions of parameters and
  performing tremendous computation, imposing significant hardware demands. In
  our paper, we propose a Dynamic Early-Exit Framework for Robotic
  Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically
  adjusts the size of the activated MLLM based on each situation at hand. The
  approach leverages a multi-exit architecture in MLLMs, which allows the model
  to terminate processing once a proper size of the model has been activated for
  a specific situation, thus avoiding further redundant computation.
  Additionally, we develop novel algorithms that establish early-termination
  criteria for DeeR, conditioned on predefined demands such as average
  computational cost (i.e., power consumption), as well as peak computational
  consumption (i.e., latency) and GPU memory usage. These enhancements ensure
  that DeeR operates efficiently under varying resource constraints while
  maintaining competitive performance. On the CALVIN robot manipulation
  benchmark, DeeR demonstrates significant reductions in computational costs of
  LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
paper_authors._id: 672ad61870734f59f1bdb3ea
paper_authors.name: Yang Yue
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-07T08:10:13.804Z
paper_authors._id: 649d475111592b1a765ac1a3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649d475111592b1a765ac1a3/rjORJjErJq-mthghan08U.jpeg
paper_authors.fullname: Yang Yue
paper_authors.type: user
paper_authors.user: Yang130
paper_authors._id: 672ad61870734f59f1bdb3eb
paper_authors.name: Yulin Wang
paper_authors._id: 672ad61870734f59f1bdb3ec
paper_authors.name: Bingyi Kang
paper_authors._id: 672ad61870734f59f1bdb3ed
paper_authors.name: Yizeng Han
paper_authors._id: 672ad61870734f59f1bdb3ee
paper_authors.name: Shenzhi Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:37:13.354Z
paper_authors._id: 6486dde1f74857df3f1a5828
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6486dde1f74857df3f1a5828/FgE80CpalBO5qqArdfwxA.jpeg
paper_authors.fullname: Shenzhi Wang
paper_authors.type: user
paper_authors.user: shenzhi-wang
paper_authors._id: 672ad61870734f59f1bdb3ef
paper_authors.name: Shiji Song
paper_authors._id: 672ad61870734f59f1bdb3f0
paper_authors.name: Jiashi Feng
paper_authors._id: 672ad61870734f59f1bdb3f1
paper_authors.name: Gao Huang
paper_title: DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for
    Efficient Robot Execution
paper_summary: MLLMs have demonstrated remarkable comprehension and reasoning capabilities
  with complex language and visual data. These advances have spurred the vision
  of establishing a generalist robotic MLLM proficient in understanding complex
  human instructions and accomplishing various embodied tasks. However,
  developing MLLMs for real-world robots is challenging due to the typically
  limited computation and memory capacities available on robotic platforms. In
  contrast, the inference of MLLMs involves storing billions of parameters and
  performing tremendous computation, imposing significant hardware demands. In
  our paper, we propose a Dynamic Early-Exit Framework for Robotic
  Vision-Language-Action Model (DeeR-VLA, or simply DeeR) that automatically
  adjusts the size of the activated MLLM based on each situation at hand. The
  approach leverages a multi-exit architecture in MLLMs, which allows the model
  to terminate processing once a proper size of the model has been activated for
  a specific situation, thus avoiding further redundant computation.
  Additionally, we develop novel algorithms that establish early-termination
  criteria for DeeR, conditioned on predefined demands such as average
  computational cost (i.e., power consumption), as well as peak computational
  consumption (i.e., latency) and GPU memory usage. These enhancements ensure
  that DeeR operates efficiently under varying resource constraints while
  maintaining competitive performance. On the CALVIN robot manipulation
  benchmark, DeeR demonstrates significant reductions in computational costs of
  LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  Code and checkpoints are available at https://github.com/yueyang130/DeeR-VLA.
paper_ai_summary: A Dynamic Early-Exit Framework for Robotic Vision-Language-Action Models optimizes computation and memory usage for MLLMs in robotics without sacrificing performance.
paper_ai_keywords: MLLMs Dynamic Early-Exit Framework DeeR-VLA multi-exit architecture early-termination criteria CALVIN robot manipulation benchmark