IllumiCraft: Unified Geometry and Illumination Diffusion for
  Controllable Video Generation
Although diffusion-based models can generate high-quality and high-resolution
video sequences from textual or image inputs, they lack explicit integration of
geometric cues when controlling scene lighting and visual appearance across
frames. To address this limitation, we propose IllumiCraft, an end-to-end
diffusion framework accepting three complementary inputs: (1)
high-dynamic-range (HDR) video maps for detailed lighting control; (2)
synthetically relit frames with randomized illumination changes (optionally
paired with a static background reference image) to provide appearance cues;
and (3) 3D point tracks that capture precise 3D geometry information. By
integrating the lighting, appearance, and geometry cues within a unified
diffusion architecture, IllumiCraft generates temporally coherent videos
aligned with user-defined prompts. It supports background-conditioned and
text-conditioned video relighting and provides better fidelity than existing
controllable video generation methods. Project Page:
https://yuanze-lin.me/IllumiCraft_page
68417439723213944b275699
Yuanze Lin
claimed_verified
2025-06-05T15:08:30.655Z
684178483b8c48d81babcbf5
https://cdn-avatars.huggingface.co/v1/production/uploads/684178483b8c48d81babcbf5/cFmEFnm_CPaqxb7yw17zv.jpeg
Yuanze Lin
user
YuanzeLin
68417439723213944b27569a
Yi-Wen Chen
admin_assigned
2025-06-05T12:52:20.026Z
65d5b580dd06ca7bac2364bf
/avatars/f380d2846507a1936eed6d2903e0d460.svg
Yi-Wen Chen
user
wenz116
68417439723213944b27569b
Yi-Hsuan Tsai
admin_assigned
2025-06-05T12:52:13.475Z
67d3a457faaad4ed2df4aba8
/avatars/fb74dff4aeb3f84b8c9862aa73e4620b.svg
Tsai Yi-Hsuan
user
Jodiiiiie
68417439723213944b27569c
Ronald Clark
68417439723213944b27569d
Ming-Hsuan Yang
IllumiCraft: Unified Geometry and Illumination Diffusion for
  Controllable Video Generation
Although diffusion-based models can generate high-quality and high-resolution
video sequences from textual or image inputs, they lack explicit integration of
geometric cues when controlling scene lighting and visual appearance across
frames. To address this limitation, we propose IllumiCraft, an end-to-end
diffusion framework accepting three complementary inputs: (1)
high-dynamic-range (HDR) video maps for detailed lighting control; (2)
synthetically relit frames with randomized illumination changes (optionally
paired with a static background reference image) to provide appearance cues;
and (3) 3D point tracks that capture precise 3D geometry information. By
integrating the lighting, appearance, and geometry cues within a unified
diffusion architecture, IllumiCraft generates temporally coherent videos
aligned with user-defined prompts. It supports background-conditioned and
text-conditioned video relighting and provides better fidelity than existing
controllable video generation methods. Project Page:
https://yuanze-lin.me/IllumiCraft_page
IllumiCraft integrates geometric cues in a diffusion framework to generate high-fidelity, temporally coherent videos from textual or image inputs.
diffusion-based models HDR video maps synthetically relit frames 3D point tracks unified diffusion architecture temporally coherent videos background-conditioned text-conditioned video relighting