title: IllumiCraft: Unified Geometry and Illumination Diffusion for
    Controllable Video Generation
summary: Although diffusion-based models can generate high-quality and high-resolution
  video sequences from textual or image inputs, they lack explicit integration of
  geometric cues when controlling scene lighting and visual appearance across
  frames. To address this limitation, we propose IllumiCraft, an end-to-end
  diffusion framework accepting three complementary inputs: (1)
  high-dynamic-range (HDR) video maps for detailed lighting control; (2)
  synthetically relit frames with randomized illumination changes (optionally
  paired with a static background reference image) to provide appearance cues;
  and (3) 3D point tracks that capture precise 3D geometry information. By
  integrating the lighting, appearance, and geometry cues within a unified
  diffusion architecture, IllumiCraft generates temporally coherent videos
  aligned with user-defined prompts. It supports background-conditioned and
  text-conditioned video relighting and provides better fidelity than existing
  controllable video generation methods. Project Page:
  https://yuanze-lin.me/IllumiCraft_page
paper_authors._id: 68417439723213944b275699
paper_authors.name: Yuanze Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-05T15:08:30.655Z
paper_authors._id: 684178483b8c48d81babcbf5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/684178483b8c48d81babcbf5/cFmEFnm_CPaqxb7yw17zv.jpeg
paper_authors.fullname: Yuanze Lin
paper_authors.type: user
paper_authors.user: YuanzeLin
paper_authors._id: 68417439723213944b27569a
paper_authors.name: Yi-Wen Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:52:20.026Z
paper_authors._id: 65d5b580dd06ca7bac2364bf
paper_authors.avatarUrl: /avatars/f380d2846507a1936eed6d2903e0d460.svg
paper_authors.fullname: Yi-Wen Chen
paper_authors.type: user
paper_authors.user: wenz116
paper_authors._id: 68417439723213944b27569b
paper_authors.name: Yi-Hsuan Tsai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:52:13.475Z
paper_authors._id: 67d3a457faaad4ed2df4aba8
paper_authors.avatarUrl: /avatars/fb74dff4aeb3f84b8c9862aa73e4620b.svg
paper_authors.fullname: Tsai Yi-Hsuan
paper_authors.type: user
paper_authors.user: Jodiiiiie
paper_authors._id: 68417439723213944b27569c
paper_authors.name: Ronald Clark
paper_authors._id: 68417439723213944b27569d
paper_authors.name: Ming-Hsuan Yang
paper_title: IllumiCraft: Unified Geometry and Illumination Diffusion for
    Controllable Video Generation
paper_summary: Although diffusion-based models can generate high-quality and high-resolution
  video sequences from textual or image inputs, they lack explicit integration of
  geometric cues when controlling scene lighting and visual appearance across
  frames. To address this limitation, we propose IllumiCraft, an end-to-end
  diffusion framework accepting three complementary inputs: (1)
  high-dynamic-range (HDR) video maps for detailed lighting control; (2)
  synthetically relit frames with randomized illumination changes (optionally
  paired with a static background reference image) to provide appearance cues;
  and (3) 3D point tracks that capture precise 3D geometry information. By
  integrating the lighting, appearance, and geometry cues within a unified
  diffusion architecture, IllumiCraft generates temporally coherent videos
  aligned with user-defined prompts. It supports background-conditioned and
  text-conditioned video relighting and provides better fidelity than existing
  controllable video generation methods. Project Page:
  https://yuanze-lin.me/IllumiCraft_page
paper_ai_summary: IllumiCraft integrates geometric cues in a diffusion framework to generate high-fidelity, temporally coherent videos from textual or image inputs.
paper_ai_keywords: diffusion-based models HDR video maps synthetically relit frames 3D point tracks unified diffusion architecture temporally coherent videos background-conditioned text-conditioned video relighting