title: Free-form language-based robotic reasoning and grasping
summary: Performing robotic grasping from a cluttered bin based on human instructions
  is a challenging task, as it requires understanding both the nuances of
  free-form language and the spatial relationships between objects.
  Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
  demonstrated remarkable reasoning capabilities across both text and images. But
  can they truly be used for this task in a zero-shot setting? And what are their
  limitations? In this paper, we explore these research questions via the
  free-form language-based robotic grasping task, and propose a novel method,
  FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
  human instructions and object spatial arrangements. Our method detects all
  objects as keypoints and uses these keypoints to annotate marks on images,
  aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
  method to determine whether a requested object is directly graspable or if
  other objects must be grasped and removed first. Since no existing dataset is
  specifically designed for this task, we introduce a synthetic dataset
  FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
  instructions and ground-truth grasping sequences. We conduct extensive analyses
  with both FreeGraspData and real-world validation with a gripper-equipped
  robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
  execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
paper_authors._id: 67d948f741d31cc626fb0adb
paper_authors.name: Runyu Jiao
paper_authors._id: 67d948f741d31cc626fb0adc
paper_authors.name: Alice Fasoli
paper_authors._id: 67d948f741d31cc626fb0add
paper_authors.name: Francesco Giuliari
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T11:33:47.178Z
paper_authors._id: 6331d79818711776b469b4ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6331d79818711776b469b4ca/g8o6s_o3_YpO5wOS7vvOC.jpeg
paper_authors.fullname: Francesco Giuliari
paper_authors.type: user
paper_authors.user: FGiuliari
paper_authors._id: 67d948f741d31cc626fb0ade
paper_authors.name: Matteo Bortolon
paper_authors._id: 67d948f741d31cc626fb0adf
paper_authors.name: Sergio Povoli
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T11:33:44.457Z
paper_authors._id: 64febddf57a40de19101c302
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64febddf57a40de19101c302/_XWwNUV5QBL2PzRhOLaKJ.jpeg
paper_authors.fullname: Sergio Povoli
paper_authors.type: user
paper_authors.user: SPovoli
paper_authors._id: 67d948f741d31cc626fb0ae0
paper_authors.name: Guofeng Mei
paper_authors._id: 67d948f741d31cc626fb0ae1
paper_authors.name: Yiming Wang
paper_authors._id: 67d948f741d31cc626fb0ae2
paper_authors.name: Fabio Poiesi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T14:57:58.696Z
paper_authors._id: 65762418769f3ee9bd5f223f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65762418769f3ee9bd5f223f/pk1g2ne31RMDVsPhmF-UQ.jpeg
paper_authors.fullname: fabio poiesi
paper_authors.type: user
paper_authors.user: fpoiesi
paper_title: Free-form language-based robotic reasoning and grasping
paper_summary: Performing robotic grasping from a cluttered bin based on human instructions
  is a challenging task, as it requires understanding both the nuances of
  free-form language and the spatial relationships between objects.
  Vision-Language Models (VLMs) trained on web-scale data, such as GPT-4o, have
  demonstrated remarkable reasoning capabilities across both text and images. But
  can they truly be used for this task in a zero-shot setting? And what are their
  limitations? In this paper, we explore these research questions via the
  free-form language-based robotic grasping task, and propose a novel method,
  FreeGrasp, leveraging the pre-trained VLMs' world knowledge to reason about
  human instructions and object spatial arrangements. Our method detects all
  objects as keypoints and uses these keypoints to annotate marks on images,
  aiming to facilitate GPT-4o's zero-shot spatial reasoning. This allows our
  method to determine whether a requested object is directly graspable or if
  other objects must be grasped and removed first. Since no existing dataset is
  specifically designed for this task, we introduce a synthetic dataset
  FreeGraspData by extending the MetaGraspNetV2 dataset with human-annotated
  instructions and ground-truth grasping sequences. We conduct extensive analyses
  with both FreeGraspData and real-world validation with a gripper-equipped
  robotic arm, demonstrating state-of-the-art performance in grasp reasoning and
  execution. Project website: https://tev-fbk.github.io/FreeGrasp/.
paper_ai_keywords: Vision-Language Models (VLMs) GPT-4o free-form language zero-shot setting spatial reasoning keypoints synthetic dataset FreeGraspData MetaGraspNetV2 human-annotated instructions ground-truth grasping sequences