title: UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video
summary: We show how to build a model that allows realistic, free-viewpoint renderings
  of a scene under novel lighting conditions from video. Our method -- UrbanIR:
  Urban Scene Inverse Rendering -- computes an inverse graphics representation
  from the video. UrbanIR jointly infers shape, albedo, visibility, and sun and
  sky illumination from a single video of unbounded outdoor scenes with unknown
  lighting. UrbanIR uses videos from cameras mounted on cars (in contrast to many
  views of the same points in typical NeRF-style estimation). As a result,
  standard methods produce poor geometry estimates (for example, roofs), and
  there are numerous ''floaters''. Errors in inverse graphics inference can
  result in strong rendering artifacts. UrbanIR uses novel losses to control
  these and other sources of error. UrbanIR uses a novel loss to make very good
  estimates of shadow volumes in the original scene. The resulting
  representations facilitate controllable editing, delivering photorealistic
  free-viewpoint renderings of relit scenes and inserted objects. Qualitative
  evaluation demonstrates strong improvements over the state-of-the-art.
paper_authors._id: 648bc91aded16248631ede5c
paper_authors.name: Zhi-Hao Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T17:02:03.385Z
paper_authors._id: 632a340c5f2ff1958c014191
paper_authors.avatarUrl: /avatars/d51ccb2e2b3e9756dea3d06b6e62f438.svg
paper_authors.fullname: Zhi-Hao Lin
paper_authors.type: user
paper_authors.user: j1a0m0e4s
paper_authors._id: 648bc91aded16248631ede5d
paper_authors.name: Bohan Liu
paper_authors._id: 648bc91aded16248631ede5e
paper_authors.name: Yi-Ting Chen
paper_authors._id: 648bc91aded16248631ede5f
paper_authors.name: David Forsyth
paper_authors._id: 648bc91aded16248631ede60
paper_authors.name: Jia-Bin Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T16:49:05.020Z
paper_authors._id: 641c139b73296f7ee256970c
paper_authors.avatarUrl: /avatars/5a2550d95e686640242840ad3bd0e680.svg
paper_authors.fullname: Jiabin Huang
paper_authors.type: user
paper_authors.user: YellowAddice
paper_authors._id: 648bc91aded16248631ede61
paper_authors.name: Anand Bhattad
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-16T17:04:59.622Z
paper_authors._id: 6303d5924ec2dfa82a5627de
paper_authors.avatarUrl: /avatars/1e433b7fb9d30b9f42dec487d658b1e3.svg
paper_authors.fullname: Anand Bhattad
paper_authors.type: user
paper_authors.user: anandbhattad
paper_authors._id: 648bc91aded16248631ede62
paper_authors.name: Shenlong Wang
paper_title: UrbanIR: Large-Scale Urban Scene Inverse Rendering from a Single Video
paper_summary: We show how to build a model that allows realistic, free-viewpoint renderings
  of a scene under novel lighting conditions from video. Our method -- UrbanIR:
  Urban Scene Inverse Rendering -- computes an inverse graphics representation
  from the video. UrbanIR jointly infers shape, albedo, visibility, and sun and
  sky illumination from a single video of unbounded outdoor scenes with unknown
  lighting. UrbanIR uses videos from cameras mounted on cars (in contrast to many
  views of the same points in typical NeRF-style estimation). As a result,
  standard methods produce poor geometry estimates (for example, roofs), and
  there are numerous ''floaters''. Errors in inverse graphics inference can
  result in strong rendering artifacts. UrbanIR uses novel losses to control
  these and other sources of error. UrbanIR uses a novel loss to make very good
  estimates of shadow volumes in the original scene. The resulting
  representations facilitate controllable editing, delivering photorealistic
  free-viewpoint renderings of relit scenes and inserted objects. Qualitative
  evaluation demonstrates strong improvements over the state-of-the-art.
paper_ai_keywords: inverse rendering free-viewpoint rendering novel lighting conditions inverse graphics representation shape inference albedo inference visibility inference illumination inference sun and sky illumination outdoor scenes cameras mounted on cars geometry estimates floaters inverse graphics inference rendering artifacts novel losses shadow volumes photorealistic rendering inserted objects controllable editing