AP-VLM: Active Perception Enabled by Vision-Language Models
<|reference_start|>AP-VLM: Active Perception Enabled by Vision-Language Models: Active perception enables robots to dynamically gather information by adjusting their viewpoints, a crucial capability for interacting with complex, partially observable environments. In this paper, we present AP-VLM, a novel framework that combines active perception with a Vision-Language Model (VLM) to guide robotic exploration and answer semantic queries. Using a 3D virtual grid overlaid on the scene and orientation adjustments, AP-VLM allows a robotic manipulator to intelligently select optimal viewpoints and orientations to resolve challenging tasks, such as identifying objects in occluded or inclined positions. We evaluate our system on two robotic platforms: a 7-DOF Franka Panda and a 6-DOF UR5, across various scenes with differing object configurations. Our results demonstrate that AP-VLM significantly outperforms passive perception methods and baseline models, including Toward Grounded Common Sense Reasoning (TGCSR), particularly in scenarios where fixed camera views are inadequate. The adaptability of AP-VLM in real-world settings shows promise for enhancing robotic systems' understanding of complex environments, bridging the gap between high-level semantic reasoning and low-level control.<|reference_end|>
@article{sripada2024ap-vlm:,
    title={AP-VLM: Active Perception Enabled by Vision-Language Models},
    author={Venkatesh Sripada, Samuel Carter, Frank Guerin, Amir Ghalamzan},
    journal={arXiv preprint arXiv:2409.17641},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.17641},
    primaryClass={cs.RO}
}