Free-form language-based robotic reasoning and grasping
Performing robotic grasping from a cluttered bin based on human instructions
is a challenging task, as it requires understanding both the nuances of
free-form language and the spatial relationships between objects.
Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
demonstrated remarkable reasoning capabilities across both text and images. But
can they truly be used for this task in a zero-shot setting? And what are their
limitations? In this paper, we explore these research questions via the
free-form language-based robotic grasping task, and propose a novel method,
FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
human instructions and object spatial arrangements. Our method detects all
objects as keypoints and uses these keypoints to annotate marks on images,
aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
method to determine whether a requested object is directly graspable or if
other objects must be grasped and removed first. Since no existing dataset is
specifically designed for this task, we introduce a synthetic dataset
FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
instructions and ground-truth grasping sequences. We conduct extensive analyses
with both FreeGraspData and real-world validation with a gripper-equipped
robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
67d948f741d31cc626fb0adb
Runyu Jiao
claimed_verified
2025-03-28T08:38:22.954Z
67d94d820d03533153d0c398
/avatars/877363a96b9da49dfe3b1f3300d4ba6e.svg
Runyu Jiao
user
rjiao
67d948f741d31cc626fb0adc
Alice Fasoli
67d948f741d31cc626fb0add
Francesco Giuliari
claimed_verified
2025-03-18T11:33:47.178Z
6331d79818711776b469b4ca
https://cdn-avatars.huggingface.co/v1/production/uploads/6331d79818711776b469b4ca/g8o6s_o3_YpO5wOS7vvOC.jpeg
Francesco Giuliari
user
FGiuliari
67d948f741d31cc626fb0ade
Matteo Bortolon
67d948f741d31cc626fb0adf
Sergio Povoli
claimed_verified
2025-03-18T11:33:44.457Z
64febddf57a40de19101c302
https://cdn-avatars.huggingface.co/v1/production/uploads/64febddf57a40de19101c302/_XWwNUV5QBL2PzRhOLaKJ.jpeg
Sergio Povoli
user
SPovoli
67d948f741d31cc626fb0ae0
Guofeng Mei
67d948f741d31cc626fb0ae1
Yiming Wang
67d948f741d31cc626fb0ae2
Fabio Poiesi
claimed_verified
2025-03-18T14:57:58.696Z
65762418769f3ee9bd5f223f
https://cdn-avatars.huggingface.co/v1/production/uploads/65762418769f3ee9bd5f223f/pk1g2ne31RMDVsPhmF-UQ.jpeg
fabio poiesi
user
fpoiesi
Free-form language-based robotic reasoning and grasping
Performing robotic grasping from a cluttered bin based on human instructions
is a challenging task, as it requires understanding both the nuances of
free-form language and the spatial relationships between objects.
Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
demonstrated remarkable reasoning capabilities across both text and images. But
can they truly be used for this task in a zero-shot setting? And what are their
limitations? In this paper, we explore these research questions via the
free-form language-based robotic grasping task, and propose a novel method,
FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
human instructions and object spatial arrangements. Our method detects all
objects as keypoints and uses these keypoints to annotate marks on images,
aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
method to determine whether a requested object is directly graspable or if
other objects must be grasped and removed first. Since no existing dataset is
specifically designed for this task, we introduce a synthetic dataset
FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
instructions and ground-truth grasping sequences. We conduct extensive analyses
with both FreeGraspData and real-world validation with a gripper-equipped
robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
A novel method, FreeGrasp, leverages pre-trained Vision-Language Models (VLMs) to perform zero-shot robotic grasping from cluttered bins based on human instructions, demonstrating state-of-the-art performance through synthetic and real-world validation.
Vision-Language Models VLMs free-form language-based robotic grasping keypoints spatial reasoning synthetic dataset FreeGraspData MetaGraspNetV2 robotic arm grasp reasoning execution