title: Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model
    And Input View Curation
summary: Generating high-quality 3D content from text, single images, or sparse view
  images remains a challenging task with broad applications.Existing methods
  typically employ multi-view diffusion models to synthesize multi-view images,
  followed by a feed-forward process for 3D reconstruction. However, these
  approaches are often constrained by a small and fixed number of input views,
  limiting their ability to capture diverse viewpoints and, even worse, leading
  to suboptimal generation results if the synthesized views are of poor quality.
  To address these limitations, we propose Flex3D, a novel two-stage framework
  capable of leveraging an arbitrary number of high-quality input views. The
  first stage consists of a candidate view generation and curation pipeline. We
  employ a fine-tuned multi-view image diffusion model and a video diffusion
  model to generate a pool of candidate views, enabling a rich representation of
  the target 3D object. Subsequently, a view selection pipeline filters these
  views based on quality and consistency, ensuring that only the high-quality and
  reliable views are used for reconstruction. In the second stage, the curated
  views are fed into a Flexible Reconstruction Model (FlexRM), built upon a
  transformer architecture that can effectively process an arbitrary number of
  inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane
  representation, enabling efficient and detailed 3D generation. Through
  extensive exploration of design and training strategies, we optimize FlexRM to
  achieve superior performance in both reconstruction and generation tasks. Our
  results demonstrate that Flex3D achieves state-of-the-art performance, with a
  user study winning rate of over 92% in 3D generation tasks when compared to
  several of the latest feed-forward 3D generative models.
paper_authors._id: 66fcb7a57d19fc1946b9029f
paper_authors.name: Junlin Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:01:05.751Z
paper_authors._id: 636e6ee287545ca5a136b4c3
paper_authors.avatarUrl: /avatars/208d32b1202e2da210146027212dbdd3.svg
paper_authors.fullname: Junlin Han
paper_authors.type: user
paper_authors.user: Junlinh
paper_authors._id: 66fcb7a57d19fc1946b902a0
paper_authors.name: Jianyuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:00:38.894Z
paper_authors._id: 649bf403fd9cea8366d669ad
paper_authors.avatarUrl: /avatars/27bd8ca9a948ec38fee950b64f669ce3.svg
paper_authors.fullname: Jianyuan Wang
paper_authors.type: user
paper_authors.user: JianyuanWang
paper_authors._id: 66fcb7a57d19fc1946b902a1
paper_authors.name: Andrea Vedaldi
paper_authors._id: 66fcb7a57d19fc1946b902a2
paper_authors.name: Philip Torr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:00:23.357Z
paper_authors._id: 6565ed28a5ec0231cb07225f
paper_authors.avatarUrl: /avatars/7f95bba9aa7811d56eecb380827abfac.svg
paper_authors.fullname: prof philip torr
paper_authors.type: user
paper_authors.user: philiptorr
paper_authors._id: 66fcb7a57d19fc1946b902a3
paper_authors.name: Filippos Kokkinos
paper_title: Flex3D: Feed-Forward 3D Generation With Flexible Reconstruction Model
    And Input View Curation
paper_summary: Generating high-quality 3D content from text, single images, or sparse view
  images remains a challenging task with broad applications.Existing methods
  typically employ multi-view diffusion models to synthesize multi-view images,
  followed by a feed-forward process for 3D reconstruction. However, these
  approaches are often constrained by a small and fixed number of input views,
  limiting their ability to capture diverse viewpoints and, even worse, leading
  to suboptimal generation results if the synthesized views are of poor quality.
  To address these limitations, we propose Flex3D, a novel two-stage framework
  capable of leveraging an arbitrary number of high-quality input views. The
  first stage consists of a candidate view generation and curation pipeline. We
  employ a fine-tuned multi-view image diffusion model and a video diffusion
  model to generate a pool of candidate views, enabling a rich representation of
  the target 3D object. Subsequently, a view selection pipeline filters these
  views based on quality and consistency, ensuring that only the high-quality and
  reliable views are used for reconstruction. In the second stage, the curated
  views are fed into a Flexible Reconstruction Model (FlexRM), built upon a
  transformer architecture that can effectively process an arbitrary number of
  inputs. FlemRM directly outputs 3D Gaussian points leveraging a tri-plane
  representation, enabling efficient and detailed 3D generation. Through
  extensive exploration of design and training strategies, we optimize FlexRM to
  achieve superior performance in both reconstruction and generation tasks. Our
  results demonstrate that Flex3D achieves state-of-the-art performance, with a
  user study winning rate of over 92% in 3D generation tasks when compared to
  several of the latest feed-forward 3D generative models.
paper_ai_summary: Flex3D generates high-quality 3D content from diverse input views using a two-stage framework with image and video diffusion models, followed by a transformer-based flexible reconstruction model.
paper_ai_keywords: multi-view diffusion models video diffusion model candidate view generation view selection pipeline Flexible Reconstruction Model FlexRM transformer architecture 3D Gaussian points tri-plane representation