title: Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion
    Models with RL Finetuning
summary: Recent advancements in the text-to-3D task leverage finetuned text-to-image
  diffusion models to generate multi-view images, followed by NeRF
  reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
  suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
  training longer with SFT improves consistency, it also causes distribution
  shift, which reduces diversity and realistic details. We argue that the SFT of
  multi-view diffusion models resembles the instruction finetuning stage of the
  LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
  Essentially, RLFT methods optimize models beyond their SFT data distribution by
  using their own outputs, effectively mitigating distribution shift. To this
  end, we introduce Carve3D, a RLFT method coupled with the Multi-view
  Reconstruction Consistency (MRC) metric, to improve the consistency of
  multi-view diffusion models. To compute MRC on a set of multi-view images, we
  compare them with their corresponding renderings of the reconstructed NeRF at
  the same viewpoints. We validate the robustness of MRC with extensive
  experiments conducted under controlled inconsistency levels. We enhance the
  base RLFT algorithm to stabilize the training process, reduce distribution
  shift, and identify scaling laws. Through qualitative and quantitative
  experiments, along with a user study, we demonstrate Carve3D's improved
  multi-view consistency, the resulting superior NeRF reconstruction quality, and
  minimal distribution shift compared to longer SFT. Project webpage:
  https://desaixie.github.io/carve-3d.
paper_authors._id: 658500ddbd67a029295a5dfa
paper_authors.name: Desai Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T07:35:48.818Z
paper_authors._id: 6279f597812ee439d9c8de67
paper_authors.avatarUrl: /avatars/49bea87294118e18df7b774f716af81a.svg
paper_authors.fullname: Desai Xie
paper_authors.type: user
paper_authors.user: desaix
paper_authors._id: 658500ddbd67a029295a5dfb
paper_authors.name: Jiahao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:08:47.270Z
paper_authors._id: 630bffcf8b327c7b8b9dfcce
paper_authors.avatarUrl: /avatars/12691f869f01217f7b92a326c30716b1.svg
paper_authors.fullname: Jiahao Li
paper_authors.type: user
paper_authors.user: 54jiahao
paper_authors._id: 658500ddbd67a029295a5dfc
paper_authors.name: Hao Tan
paper_authors._id: 658500ddbd67a029295a5dfd
paper_authors.name: Xin Sun
paper_authors._id: 658500ddbd67a029295a5dfe
paper_authors.name: Zhixin Shu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-22T12:09:53.819Z
paper_authors._id: 62a8efd508a7ea93ff18785a
paper_authors.avatarUrl: /avatars/ff259233d437833a304329bb973a5a04.svg
paper_authors.fullname: Zhixin Shu
paper_authors.type: user
paper_authors.user: zhixinshu
paper_authors._id: 658500ddbd67a029295a5dff
paper_authors.name: Yi Zhou
paper_authors._id: 658500ddbd67a029295a5e00
paper_authors.name: Sai Bi
paper_authors._id: 658500ddbd67a029295a5e01
paper_authors.name: Sören Pirk
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-29T09:00:54.574Z
paper_authors._id: 658e87be43edad2169db050a
paper_authors.avatarUrl: /avatars/467d8949626539a7482dccbee4fb4d25.svg
paper_authors.fullname: Soren Pirk
paper_authors.type: user
paper_authors.user: pirk
paper_authors._id: 658500ddbd67a029295a5e02
paper_authors.name: Arie E. Kaufman
paper_title: Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion
    Models with RL Finetuning
paper_summary: Recent advancements in the text-to-3D task leverage finetuned text-to-image
  diffusion models to generate multi-view images, followed by NeRF
  reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
  suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
  training longer with SFT improves consistency, it also causes distribution
  shift, which reduces diversity and realistic details. We argue that the SFT of
  multi-view diffusion models resembles the instruction finetuning stage of the
  LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
  Essentially, RLFT methods optimize models beyond their SFT data distribution by
  using their own outputs, effectively mitigating distribution shift. To this
  end, we introduce Carve3D, a RLFT method coupled with the Multi-view
  Reconstruction Consistency (MRC) metric, to improve the consistency of
  multi-view diffusion models. To compute MRC on a set of multi-view images, we
  compare them with their corresponding renderings of the reconstructed NeRF at
  the same viewpoints. We validate the robustness of MRC with extensive
  experiments conducted under controlled inconsistency levels. We enhance the
  base RLFT algorithm to stabilize the training process, reduce distribution
  shift, and identify scaling laws. Through qualitative and quantitative
  experiments, along with a user study, we demonstrate Carve3D's improved
  multi-view consistency, the resulting superior NeRF reconstruction quality, and
  minimal distribution shift compared to longer SFT. Project webpage:
  https://desaixie.github.io/carve-3d.
paper_ai_summary: Carve3D, integrating RLFT with a Multi-view Reconstruction Consistency (MRC) metric, enhances multi-view consistency and NeRF reconstruction quality while mitigating distribution shift in text-to-3D models.
paper_ai_keywords: text-to-3D diffusion models NeRF supervised finetuning (SFT) RL finetuning (RLFT) Multi-view Reconstruction Consistency (MRC) distribution shift base RLFT algorithm scaling laws qualitative and quantitative experiments user study