title: PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
    Prediction
summary: We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
  a 3D object from a few unposed images even with little visual overlap, while
  simultaneously estimating the relative camera poses in ~1.3 seconds on a single
  A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
  blocks to exchange information between 3D object tokens and 2D image tokens; we
  predict a coarse point cloud for each view, and then use a differentiable
  Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
  amount of multi-view posed data of ~1M objects, PF-LRM shows strong
  cross-dataset generalization ability, and outperforms baseline methods by a
  large margin in terms of pose prediction accuracy and 3D reconstruction quality
  on various unseen evaluation datasets. We also demonstrate our model's
  applicability in downstream text/image-to-3D task with fast feed-forward
  inference. Our project website is at: https://totoro97.github.io/pf-lrm .
paper_authors._id: 655d9a35ba94da7590e78cef
paper_authors.name: Peng Wang
paper_authors._id: 655d9a35ba94da7590e78cf0
paper_authors.name: Hao Tan
paper_authors._id: 655d9a35ba94da7590e78cf1
paper_authors.name: Sai Bi
paper_authors._id: 655d9a35ba94da7590e78cf2
paper_authors.name: Yinghao Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:12:15.296Z
paper_authors._id: 6555da9adcf410fd0753569c
paper_authors.avatarUrl: /avatars/ac58a796bb54f334fdb475a4b75c4d27.svg
paper_authors.fullname: Yinghao Xu
paper_authors.type: user
paper_authors.user: justimyhxu
paper_authors._id: 655d9a35ba94da7590e78cf3
paper_authors.name: Fujun Luan
paper_authors._id: 655d9a35ba94da7590e78cf4
paper_authors.name: Kalyan Sunkavalli
paper_authors._id: 655d9a35ba94da7590e78cf5
paper_authors.name: Wenping Wang
paper_authors._id: 655d9a35ba94da7590e78cf6
paper_authors.name: Zexiang Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-22T13:12:55.585Z
paper_authors._id: 65565b1b8616d8d740abb976
paper_authors.avatarUrl: /avatars/c245b19ed05292bf1bf4a9d8cf2a409c.svg
paper_authors.fullname: Zexiang Xu
paper_authors.type: user
paper_authors.user: zexiangxu
paper_authors._id: 655d9a35ba94da7590e78cf7
paper_authors.name: Kai Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-23T10:49:11.922Z
paper_authors._id: 6351a882a4845f26d16a28d7
paper_authors.avatarUrl: /avatars/4f8629fbf22faeb6511f4068f2828809.svg
paper_authors.fullname: Kai Zhang
paper_authors.type: user
paper_authors.user: Kai-46
paper_title: PF-LRM: Pose-Free Large Reconstruction Model for Joint Pose and Shape
    Prediction
paper_summary: We propose a Pose-Free Large Reconstruction Model (PF-LRM) for reconstructing
  a 3D object from a few unposed images even with little visual overlap, while
  simultaneously estimating the relative camera poses in ~1.3 seconds on a single
  A100 GPU. PF-LRM is a highly scalable method utilizing the self-attention
  blocks to exchange information between 3D object tokens and 2D image tokens; we
  predict a coarse point cloud for each view, and then use a differentiable
  Perspective-n-Point (PnP) solver to obtain camera poses. When trained on a huge
  amount of multi-view posed data of ~1M objects, PF-LRM shows strong
  cross-dataset generalization ability, and outperforms baseline methods by a
  large margin in terms of pose prediction accuracy and 3D reconstruction quality
  on various unseen evaluation datasets. We also demonstrate our model's
  applicability in downstream text/image-to-3D task with fast feed-forward
  inference. Our project website is at: https://totoro97.github.io/pf-lrm .
paper_ai_summary: A Pose-Free Large Reconstruction Model using self-attention for 3D reconstruction from few unposed images, estimating camera poses, and generalizing across datasets.
paper_ai_keywords: Pose-Free Large Reconstruction Model self-attention 3D object tokens 2D image tokens Perspective-n-Point PnP solver cross-dataset generalization text-to-3D image-to-3D feed-forward inference