Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion
  Models with RL Finetuning
Recent advancements in the text-to-3D task leverage finetuned text-to-image
diffusion models to generate multi-view images, followed by NeRF
reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
training longer with SFT improves consistency, it also causes distribution
shift, which reduces diversity and realistic details. We argue that the SFT of
multi-view diffusion models resembles the instruction finetuning stage of the
LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
Essentially, RLFT methods optimize models beyond their SFT data distribution by
using their own outputs, effectively mitigating distribution shift. To this
end, we introduce Carve3D, a RLFT method coupled with the Multi-view
Reconstruction Consistency (MRC) metric, to improve the consistency of
multi-view diffusion models. To compute MRC on a set of multi-view images, we
compare them with their corresponding renderings of the reconstructed NeRF at
the same viewpoints. We validate the robustness of MRC with extensive
experiments conducted under controlled inconsistency levels. We enhance the
base RLFT algorithm to stabilize the training process, reduce distribution
shift, and identify scaling laws. Through qualitative and quantitative
experiments, along with a user study, we demonstrate Carve3D's improved
multi-view consistency, the resulting superior NeRF reconstruction quality, and
minimal distribution shift compared to longer SFT. Project webpage:
https://desaixie.github.io/carve-3d.
658500ddbd67a029295a5dfa
Desai Xie
claimed_verified
2023-12-22T07:35:48.818Z
6279f597812ee439d9c8de67
/avatars/49bea87294118e18df7b774f716af81a.svg
Desai Xie
user
desaix
658500ddbd67a029295a5dfb
Jiahao Li
admin_assigned
2023-12-22T12:08:47.270Z
630bffcf8b327c7b8b9dfcce
/avatars/12691f869f01217f7b92a326c30716b1.svg
Jiahao Li
user
54jiahao
658500ddbd67a029295a5dfc
Hao Tan
658500ddbd67a029295a5dfd
Xin Sun
658500ddbd67a029295a5dfe
Zhixin Shu
admin_assigned
2023-12-22T12:09:53.819Z
62a8efd508a7ea93ff18785a
/avatars/ff259233d437833a304329bb973a5a04.svg
Zhixin Shu
user
zhixinshu
658500ddbd67a029295a5dff
Yi Zhou
658500ddbd67a029295a5e00
Sai Bi
658500ddbd67a029295a5e01
Sören Pirk
claimed_verified
2023-12-29T09:00:54.574Z
658e87be43edad2169db050a
/avatars/467d8949626539a7482dccbee4fb4d25.svg
Soren Pirk
user
pirk
658500ddbd67a029295a5e02
Arie E. Kaufman
Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion
  Models with RL Finetuning
Recent advancements in the text-to-3D task leverage finetuned text-to-image
diffusion models to generate multi-view images, followed by NeRF
reconstruction. Yet, existing supervised finetuned (SFT) diffusion models still
suffer from multi-view inconsistency and the resulting NeRF artifacts. Although
training longer with SFT improves consistency, it also causes distribution
shift, which reduces diversity and realistic details. We argue that the SFT of
multi-view diffusion models resembles the instruction finetuning stage of the
LLM alignment pipeline and can benefit from RL finetuning (RLFT) methods.
Essentially, RLFT methods optimize models beyond their SFT data distribution by
using their own outputs, effectively mitigating distribution shift. To this
end, we introduce Carve3D, a RLFT method coupled with the Multi-view
Reconstruction Consistency (MRC) metric, to improve the consistency of
multi-view diffusion models. To compute MRC on a set of multi-view images, we
compare them with their corresponding renderings of the reconstructed NeRF at
the same viewpoints. We validate the robustness of MRC with extensive
experiments conducted under controlled inconsistency levels. We enhance the
base RLFT algorithm to stabilize the training process, reduce distribution
shift, and identify scaling laws. Through qualitative and quantitative
experiments, along with a user study, we demonstrate Carve3D's improved
multi-view consistency, the resulting superior NeRF reconstruction quality, and
minimal distribution shift compared to longer SFT. Project webpage:
https://desaixie.github.io/carve-3d.
Carve3D, integrating RLFT with a Multi-view Reconstruction Consistency (MRC) metric, enhances multi-view consistency and NeRF reconstruction quality while mitigating distribution shift in text-to-3D models.
text-to-3D diffusion models NeRF supervised finetuning (SFT) RL finetuning (RLFT) Multi-view Reconstruction Consistency (MRC) distribution shift base RLFT algorithm scaling laws qualitative and quantitative experiments user study