What Makes for Text to 360-degree Panorama Generation with Stable
  Diffusion?
Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion,
has stimulated research to adapt them to 360-degree panorama generation. Prior
work has demonstrated the feasibility of using conventional low-rank adaptation
techniques on pre-trained diffusion models to generate panoramic images.
However, the substantial domain gap between perspective and panoramic images
raises questions about the underlying mechanisms enabling this empirical
success. We hypothesize and examine that the trainable counterparts exhibit
distinct behaviors when fine-tuned on panoramic data, and such an adaptation
conceals some intrinsic mechanism to leverage the prior knowledge within the
pre-trained diffusion models. Our analysis reveals the following: 1) the query
and key matrices in the attention modules are responsible for common
information that can be shared between the panoramic and perspective domains,
thus are less relevant to panorama generation; and 2) the value and output
weight matrices specialize in adapting pre-trained knowledge to the panoramic
domain, playing a more critical role during fine-tuning for panorama
generation. We empirically verify these insights by introducing a simple
framework called UniPano, with the objective of establishing an elegant
baseline for future research. UniPano not only outperforms existing methods but
also significantly reduces memory usage and training time compared to prior
dual-branch approaches, making it scalable for end-to-end panorama generation
with higher resolution. The code will be released.
6837dae6e9b21653755a05d4
Jinhong Ni
claimed_verified
2025-05-29T07:42:03.476Z
64c71a5647418a0a59e5c7cb
/avatars/a99ab24c0c19b1399d2e6795fb9d7000.svg
Jinhong Ni
user
mcleanie
6837dae6e9b21653755a05d5
Chang-Bin Zhang
claimed_verified
2025-05-29T07:41:59.930Z
65434daa5a36a8774d0e2271
/avatars/abc3ddec72072121130d581e32cd9045.svg
Allen Zhang
user
allencbzhang
6837dae6e9b21653755a05d6
Qiang Zhang
6837dae6e9b21653755a05d7
Jing Zhang
What Makes for Text to 360-degree Panorama Generation with Stable
  Diffusion?
Recent prosperity of text-to-image diffusion models, e.g. Stable Diffusion,
has stimulated research to adapt them to 360-degree panorama generation. Prior
work has demonstrated the feasibility of using conventional low-rank adaptation
techniques on pre-trained diffusion models to generate panoramic images.
However, the substantial domain gap between perspective and panoramic images
raises questions about the underlying mechanisms enabling this empirical
success. We hypothesize and examine that the trainable counterparts exhibit
distinct behaviors when fine-tuned on panoramic data, and such an adaptation
conceals some intrinsic mechanism to leverage the prior knowledge within the
pre-trained diffusion models. Our analysis reveals the following: 1) the query
and key matrices in the attention modules are responsible for common
information that can be shared between the panoramic and perspective domains,
thus are less relevant to panorama generation; and 2) the value and output
weight matrices specialize in adapting pre-trained knowledge to the panoramic
domain, playing a more critical role during fine-tuning for panorama
generation. We empirically verify these insights by introducing a simple
framework called UniPano, with the objective of establishing an elegant
baseline for future research. UniPano not only outperforms existing methods but
also significantly reduces memory usage and training time compared to prior
dual-branch approaches, making it scalable for end-to-end panorama generation
with higher resolution. The code will be released.
Analysis of fine-tuning diffusion models for panoramic image generation reveals distinct roles of attention module matrices and introduces UniPano, a memory-efficient and speed-enhanced baseline framework.
text-to-image diffusion models Stable Diffusion low-rank adaptation pre-trained diffusion models attention modules query matrices key matrices value matrices output weight matrices panoramic image generation domain gap common information pre-trained knowledge UniPano end-to-end panorama generation memory usage training time