title: FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
summary: We present FoundationPose, a unified foundation model for 6D object pose
  estimation and tracking, supporting both model-based and model-free setups. Our
  approach can be instantly applied at test-time to a novel object without
  fine-tuning, as long as its CAD model is given, or a small number of reference
  images are captured. We bridge the gap between these two setups with a neural
  implicit representation that allows for effective novel view synthesis, keeping
  the downstream pose estimation modules invariant under the same unified
  framework. Strong generalizability is achieved via large-scale synthetic
  training, aided by a large language model (LLM), a novel transformer-based
  architecture, and contrastive learning formulation. Extensive evaluation on
  multiple public datasets involving challenging scenarios and objects indicate
  our unified approach outperforms existing methods specialized for each task by
  a large margin. In addition, it even achieves comparable results to
  instance-level methods despite the reduced assumptions. Project page:
  https://nvlabs.github.io/FoundationPose/
paper_authors._id: 657a6a4ec6240db29fee60cb
paper_authors.name: Bowen Wen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-14T13:51:24.015Z
paper_authors._id: 64aeee717b3b3386662e9e62
paper_authors.avatarUrl: /avatars/8fd244261d6128a109c25ef8578e186b.svg
paper_authors.fullname: Bowen Wen
paper_authors.type: user
paper_authors.user: wenbowen
paper_authors._id: 657a6a4ec6240db29fee60cc
paper_authors.name: Wei Yang
paper_authors._id: 657a6a4ec6240db29fee60cd
paper_authors.name: Jan Kautz
paper_authors._id: 657a6a4ec6240db29fee60ce
paper_authors.name: Stan Birchfield
paper_title: FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
paper_summary: We present FoundationPose, a unified foundation model for 6D object pose
  estimation and tracking, supporting both model-based and model-free setups. Our
  approach can be instantly applied at test-time to a novel object without
  fine-tuning, as long as its CAD model is given, or a small number of reference
  images are captured. We bridge the gap between these two setups with a neural
  implicit representation that allows for effective novel view synthesis, keeping
  the downstream pose estimation modules invariant under the same unified
  framework. Strong generalizability is achieved via large-scale synthetic
  training, aided by a large language model (LLM), a novel transformer-based
  architecture, and contrastive learning formulation. Extensive evaluation on
  multiple public datasets involving challenging scenarios and objects indicate
  our unified approach outperforms existing methods specialized for each task by
  a large margin. In addition, it even achieves comparable results to
  instance-level methods despite the reduced assumptions. Project page:
  https://nvlabs.github.io/FoundationPose/
paper_ai_summary: FoundationPose, a unified foundation model using neural implicit representation and transformer architecture, outperforms specialized models in 6D object pose estimation and tracking.
paper_ai_keywords: neural implicit representation transformer-based architecture contrastive learning formulation