title: CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View
    Synthesis
summary: Satellite-to-street view synthesis aims at generating a realistic street-view
  image from its corresponding satellite-view image. Although stable diffusion
  models have exhibit remarkable performance in a variety of image generation
  applications, their reliance on similar-view inputs to control the generated
  structure or texture restricts their application to the challenging cross-view
  synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion
  model for satellite-to-street view synthesis. To address the challenges posed
  by the large discrepancy across views, we design the satellite scene structure
  estimation and cross-view texture mapping modules to construct the structural
  and textural controls for street-view image synthesis. We further design a
  cross-view control guided denoising process that incorporates the above
  controls via an enhanced cross-view attention module. To achieve a more
  comprehensive evaluation of the synthesis results, we additionally design a
  GPT-based scoring method as a supplement to standard evaluation metrics. We
  also explore the effect of different data sources (e.g., text, maps, building
  heights, and multi-temporal satellite imagery) on this task. Results on three
  public cross-view datasets show that CrossViewDiff outperforms current
  state-of-the-art on both standard and GPT-based evaluation metrics, generating
  high-quality street-view panoramas with more realistic structures and textures
  across rural, suburban, and urban scenes. The code and models of this work will
  be released at https://opendatalab.github.io/CrossViewDiff/.
paper_authors._id: 66d5b674b66342762fd54d02
paper_authors.name: Weijia Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T07:48:23.479Z
paper_authors._id: 66d5b56c77a026c3d2086a79
paper_authors.avatarUrl: /avatars/45da07fd82fd455955faa05b27a6393f.svg
paper_authors.fullname: Weijia Li
paper_authors.type: user
paper_authors.user: liweijia
paper_authors._id: 66d5b674b66342762fd54d03
paper_authors.name: Jun He
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-16T09:44:49.833Z
paper_authors._id: 670ddb69d6ac6394419d88c5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/XxnGNaX3FWug4aiZVjg93.png
paper_authors.fullname: Jun He
paper_authors.type: user
paper_authors.user: JunHe0915
paper_authors._id: 66d5b674b66342762fd54d04
paper_authors.name: Junyan Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T14:48:59.572Z
paper_authors._id: 66978ee0b8656f6506b4acb2
paper_authors.avatarUrl: /avatars/298acb8222e189fce4368985ee5374a1.svg
paper_authors.fullname: Junyan Ye
paper_authors.type: user
paper_authors.user: Yejy53
paper_authors._id: 66d5b674b66342762fd54d05
paper_authors.name: Huaping Zhong
paper_authors._id: 66d5b674b66342762fd54d06
paper_authors.name: Zhimeng Zheng
paper_authors._id: 66d5b674b66342762fd54d07
paper_authors.name: Zilong Huang
paper_authors._id: 66d5b674b66342762fd54d08
paper_authors.name: Dahua Lin
paper_authors._id: 66d5b674b66342762fd54d09
paper_authors.name: Conghui He
paper_title: CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View
    Synthesis
paper_summary: Satellite-to-street view synthesis aims at generating a realistic street-view
  image from its corresponding satellite-view image. Although stable diffusion
  models have exhibit remarkable performance in a variety of image generation
  applications, their reliance on similar-view inputs to control the generated
  structure or texture restricts their application to the challenging cross-view
  synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion
  model for satellite-to-street view synthesis. To address the challenges posed
  by the large discrepancy across views, we design the satellite scene structure
  estimation and cross-view texture mapping modules to construct the structural
  and textural controls for street-view image synthesis. We further design a
  cross-view control guided denoising process that incorporates the above
  controls via an enhanced cross-view attention module. To achieve a more
  comprehensive evaluation of the synthesis results, we additionally design a
  GPT-based scoring method as a supplement to standard evaluation metrics. We
  also explore the effect of different data sources (e.g., text, maps, building
  heights, and multi-temporal satellite imagery) on this task. Results on three
  public cross-view datasets show that CrossViewDiff outperforms current
  state-of-the-art on both standard and GPT-based evaluation metrics, generating
  high-quality street-view panoramas with more realistic structures and textures
  across rural, suburban, and urban scenes. The code and models of this work will
  be released at https://opendatalab.github.io/CrossViewDiff/.
paper_ai_summary: CrossViewDiff, a cross-view diffusion model, enhances satellite-to-street view synthesis by incorporating structural and textural controls, achieving superior results across various datasets.
paper_ai_keywords: cross-view diffusion model satellite scene structure estimation cross-view texture mapping denoising process cross-view attention module GPT-based scoring method