title: MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
    Model
summary: Open-world 3D reconstruction models have recently garnered significant
  attention. However, without sufficient 3D inductive bias, existing methods
  typically entail expensive training costs and struggle to extract high-quality
  3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
  model that explicitly leverages 3D native structure, input guidance, and
  training supervision. Specifically, instead of using a triplane representation,
  we store features in 3D sparse voxels and combine transformers with 3D
  convolutions to leverage an explicit 3D structure and projective bias. In
  addition to sparse-view RGB input, we require the network to take input and
  generate corresponding normal maps. The input normal maps can be predicted by
  2D diffusion models, significantly aiding in the guidance and refinement of the
  geometry's learning. Moreover, by combining Signed Distance Function (SDF)
  supervision with surface rendering, we directly learn to generate high-quality
  meshes without the need for complex multi-stage training processes. By
  incorporating these explicit 3D biases, MeshFormer can be trained efficiently
  and deliver high-quality textured meshes with fine-grained geometric details.
  It can also be integrated with 2D diffusion models to enable fast
  single-image-to-3D and text-to-3D tasks. Project page:
  https://meshformer3d.github.io
paper_authors._id: 66c3f562d9a7974122b19504
paper_authors.name: Minghua Liu
paper_authors._id: 66c3f562d9a7974122b19505
paper_authors.name: Chong Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:51:49.317Z
paper_authors._id: 60747cbf3ea03830676542b5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60747cbf3ea03830676542b5/wGr1Jzz520JM9nZ-UcLyb.png
paper_authors.fullname: Chong Zeng
paper_authors.type: user
paper_authors.user: NCJ
paper_authors._id: 66c3f562d9a7974122b19506
paper_authors.name: Xinyue Wei
paper_authors._id: 66c3f562d9a7974122b19507
paper_authors.name: Ruoxi Shi
paper_authors._id: 66c3f562d9a7974122b19508
paper_authors.name: Linghao Chen
paper_authors._id: 66c3f562d9a7974122b19509
paper_authors.name: Chao Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-22T19:51:46.522Z
paper_authors._id: 6468a48cff18750165a5aad2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/vmXEWka_q2a4mBDNjbMuY.jpeg
paper_authors.fullname: Chao Xu
paper_authors.type: user
paper_authors.user: chaoxu
paper_authors._id: 66c3f562d9a7974122b1950a
paper_authors.name: Mengqi Zhang
paper_authors._id: 66c3f562d9a7974122b1950b
paper_authors.name: Zhaoning Wang
paper_authors._id: 66c3f562d9a7974122b1950c
paper_authors.name: Xiaoshuai Zhang
paper_authors._id: 66c3f562d9a7974122b1950d
paper_authors.name: Isabella Liu
paper_authors._id: 66c3f562d9a7974122b1950e
paper_authors.name: Hongzhi Wu
paper_authors._id: 66c3f562d9a7974122b1950f
paper_authors.name: Hao Su
paper_title: MeshFormer: High-Quality Mesh Generation with 3D-Guided Reconstruction
    Model
paper_summary: Open-world 3D reconstruction models have recently garnered significant
  attention. However, without sufficient 3D inductive bias, existing methods
  typically entail expensive training costs and struggle to extract high-quality
  3D meshes. In this work, we introduce MeshFormer, a sparse-view reconstruction
  model that explicitly leverages 3D native structure, input guidance, and
  training supervision. Specifically, instead of using a triplane representation,
  we store features in 3D sparse voxels and combine transformers with 3D
  convolutions to leverage an explicit 3D structure and projective bias. In
  addition to sparse-view RGB input, we require the network to take input and
  generate corresponding normal maps. The input normal maps can be predicted by
  2D diffusion models, significantly aiding in the guidance and refinement of the
  geometry's learning. Moreover, by combining Signed Distance Function (SDF)
  supervision with surface rendering, we directly learn to generate high-quality
  meshes without the need for complex multi-stage training processes. By
  incorporating these explicit 3D biases, MeshFormer can be trained efficiently
  and deliver high-quality textured meshes with fine-grained geometric details.
  It can also be integrated with 2D diffusion models to enable fast
  single-image-to-3D and text-to-3D tasks. Project page:
  https://meshformer3d.github.io
paper_ai_keywords: 3D reconstruction models triplane representation 3D sparse voxels transformers 3D convolutions projective bias normal maps 2D diffusion models Signed Distance Function (SDF) surface rendering multi-stage training processes textured meshes fine-grained geometric details single-image-to-3D text-to-3D