Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation
<|reference_start|>Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation: In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.<|reference_end|>
@article{wang2024observe,
    title={Observe Then Act: Asynchronous Active Vision-Action Model for Robotic
  Manipulation},
    author={Guokang Wang, Hang Li, Shuyuan Zhang, Yanhong Liu, Huaping Liu},
    journal={arXiv preprint arXiv:2409.14891},
    year={2024},
    archivePrefix={arXiv},
    eprint={2409.14891},
    primaryClass={cs.RO cs.CV}
}