title: What Makes for Text to 360-degree Panorama Generation with Stable
    Diffusion?
summary: Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion,
  has stimulated research to adapt them to 360-degree panorama generation. Prior
  work has demonstrated the feasibility of using conventional low-rank adaptation
  techniques on pre-trained diffusion models to generate panoramic images.
  However, the substantial domain gap between perspective and panoramic images
  raises questions about the underlying mechanisms enabling this empirical
  success. We hypothesize and examine that the trainable counterparts exhibit
  distinct behaviors when fine-tuned on panoramic data, and such an adaptation
  conceals some intrinsic mechanism to leverage the prior knowledge within the
  pre-trained diffusion models. Our analysis reveals the following: 1) the query
  and key matrices in the attention modules are responsible for common
  information that can be shared between the panoramic and perspective domains,
  thus are less relevant to panorama generation; and 2) the value and output
  weight matrices specialize in adapting pre-trained knowledge to the panoramic
  domain, playing a more critical role during fine-tuning for panorama
  generation. We empirically verify these insights by introducing a simple
  framework called UniPano, with the objective of establishing an elegant
  baseline for future research. UniPano not only outperforms existing methods but
  also significantly reduces memory usage and training time compared to prior
  dual-branch approaches, making it scalable for end-to-end panorama generation
  with higher resolution. The code will be released.
paper_authors._id: 6837dae6e9b21653755a05d4
paper_authors.name: Jinhong Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:42:03.476Z
paper_authors._id: 64c71a5647418a0a59e5c7cb
paper_authors.avatarUrl: /avatars/a99ab24c0c19b1399d2e6795fb9d7000.svg
paper_authors.fullname: Jinhong Ni
paper_authors.type: user
paper_authors.user: mcleanie
paper_authors._id: 6837dae6e9b21653755a05d5
paper_authors.name: Chang-Bin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-29T07:41:59.930Z
paper_authors._id: 65434daa5a36a8774d0e2271
paper_authors.avatarUrl: /avatars/abc3ddec72072121130d581e32cd9045.svg
paper_authors.fullname: Allen Zhang
paper_authors.type: user
paper_authors.user: allencbzhang
paper_authors._id: 6837dae6e9b21653755a05d6
paper_authors.name: Qiang Zhang
paper_authors._id: 6837dae6e9b21653755a05d7
paper_authors.name: Jing Zhang
paper_title: What Makes for Text to 360-degree Panorama Generation with Stable
    Diffusion?
paper_summary: Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion,
  has stimulated research to adapt them to 360-degree panorama generation. Prior
  work has demonstrated the feasibility of using conventional low-rank adaptation
  techniques on pre-trained diffusion models to generate panoramic images.
  However, the substantial domain gap between perspective and panoramic images
  raises questions about the underlying mechanisms enabling this empirical
  success. We hypothesize and examine that the trainable counterparts exhibit
  distinct behaviors when fine-tuned on panoramic data, and such an adaptation
  conceals some intrinsic mechanism to leverage the prior knowledge within the
  pre-trained diffusion models. Our analysis reveals the following: 1) the query
  and key matrices in the attention modules are responsible for common
  information that can be shared between the panoramic and perspective domains,
  thus are less relevant to panorama generation; and 2) the value and output
  weight matrices specialize in adapting pre-trained knowledge to the panoramic
  domain, playing a more critical role during fine-tuning for panorama
  generation. We empirically verify these insights by introducing a simple
  framework called UniPano, with the objective of establishing an elegant
  baseline for future research. UniPano not only outperforms existing methods but
  also significantly reduces memory usage and training time compared to prior
  dual-branch approaches, making it scalable for end-to-end panorama generation
  with higher resolution. The code will be released.
paper_ai_summary: Analysis of fine-tuning diffusion models for panoramic image generation reveals distinct roles of attention module matrices and introduces UniPano, a memory-efficient and speed-enhanced baseline framework.
paper_ai_keywords: text-to-image diffusion models Stable Diffusion low-rank adaptation pre-trained diffusion models attention modules query matrices key matrices value matrices output weight matrices panoramic image generation domain gap common information pre-trained knowledge UniPano end-to-end panorama generation memory usage training time