villa-X: Enhancing Latent Action Modeling in Vision-Language-Action
  Models
Visual-Language-Action (VLA) models have emerged as a popular paradigm for
learning robot manipulation policies that can follow language instructions and
generalize to novel scenarios. Recent work has begun to explore the
incorporation of latent actions, an abstract representation of visual change
between two frames, into VLA pre-training. In this paper, we introduce villa-X,
a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent
action modeling for learning generalizable robot manipulation policies. Our
approach improves both how latent actions are learned and how they are
incorporated into VLA pre-training. Together, these contributions enable
villa-X to achieve superior performance across simulated environments including
SIMPLER and LIBERO, as well as on two real-world robot setups including gripper
and dexterous hand manipulation. We believe the ViLLA paradigm holds
significant promise, and that our villa-X provides a strong foundation for
future research.
688c14038c434640078cc33a
Xiaoyu Chen
688c14038c434640078cc33b
Hangxing Wei
688c14038c434640078cc33c
Pushi Zhang
688c14038c434640078cc33d
Chuheng Zhang
688c14038c434640078cc33e
Kaixin Wang
688c14038c434640078cc33f
Yanjiang Guo
688c14038c434640078cc340
Rushuai Yang
688c14038c434640078cc341
Yucen Wang
688c14038c434640078cc342
Xinquan Xiao
688c14038c434640078cc343
Li Zhao
688c14038c434640078cc344
Jianyu Chen
688c14038c434640078cc345
Jiang Bian
villa-X: Enhancing Latent Action Modeling in Vision-Language-Action
  Models
Visual-Language-Action (VLA) models have emerged as a popular paradigm for
learning robot manipulation policies that can follow language instructions and
generalize to novel scenarios. Recent work has begun to explore the
incorporation of latent actions, an abstract representation of visual change
between two frames, into VLA pre-training. In this paper, we introduce villa-X,
a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent
action modeling for learning generalizable robot manipulation policies. Our
approach improves both how latent actions are learned and how they are
incorporated into VLA pre-training. Together, these contributions enable
villa-X to achieve superior performance across simulated environments including
SIMPLER and LIBERO, as well as on two real-world robot setups including gripper
and dexterous hand manipulation. We believe the ViLLA paradigm holds
significant promise, and that our villa-X provides a strong foundation for
future research.
The ViLLA framework enhances VLA models by incorporating latent actions, improving performance in both simulated and real-world robot manipulation tasks.
Visual-Language-Action (VLA) models latent actions ViLLA framework SIMPLER LIBERO gripper manipulation dexterous hand manipulation