MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual
  Storytelling via Multi-Layered Semantic-Aware Denoising
Visual storytelling often uses nontypical aspect-ratio images like scroll
paintings, comic strips, and panoramas to create an expressive and compelling
narrative. While generative AI has achieved great success and shown the
potential to reshape the creative industry, it remains a challenge to generate
coherent and engaging content with arbitrary size and controllable style,
concept, and layout, all of which are essential for visual storytelling. To
overcome the shortcomings of previous methods including repetitive content,
style inconsistency, and lack of controllability, we propose MagicScroll, a
multi-layered, progressive diffusion-based image generation framework with a
novel semantic-aware denoising process. The model enables fine-grained control
over the generated image on object, scene, and background levels with text,
image, and layout conditions. We also establish the first benchmark for
nontypical aspect-ratio image generation for visual storytelling including
mediums like paintings, comics, and cinematic panoramas, with customized
metrics for systematic evaluation. Through comparative and ablation studies,
MagicScroll showcases promising results in aligning with the narrative text,
improving visual coherence, and engaging the audience. We plan to release the
code and benchmark in the hope of a better collaboration between AI researchers
and creative practitioners involving visual storytelling.
65812ffd4cdbd00e3de43a22
Bingyuan Wang
admin_assigned
2023-12-19T13:48:41.010Z
631c01a19edb6d320a16fe8f
/avatars/93f04a678d7d1f78decfb279767e42d0.svg
Bingyuan Wang
user
BingyuanWang
65812ffd4cdbd00e3de43a23
Hengyu Meng
admin_assigned
2023-12-19T09:37:37.088Z
63edfe1c5f1300034dd90a37
/avatars/fa742c6da029ce73b395d1a12a97753f.svg
Hyuno
user
HengyuMeng
65812ffd4cdbd00e3de43a24
Zeyu Cai
65812ffd4cdbd00e3de43a25
Lanjiong Li
65812ffd4cdbd00e3de43a26
Yue Ma
65812ffd4cdbd00e3de43a27
Qifeng Chen
admin_assigned
2023-12-19T09:38:26.050Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
65812ffd4cdbd00e3de43a28
Zeyu Wang
claimed_verified
2023-12-19T16:01:25.896Z
6581a37c7d2dbe3781d48ac9
/avatars/95211723e67434a29082bf6c29bae8e3.svg
Zeyu Wang
user
zachzeyuwang
MagicScroll: Nontypical Aspect-Ratio Image Generation for Visual
  Storytelling via Multi-Layered Semantic-Aware Denoising
Visual storytelling often uses nontypical aspect-ratio images like scroll
paintings, comic strips, and panoramas to create an expressive and compelling
narrative. While generative AI has achieved great success and shown the
potential to reshape the creative industry, it remains a challenge to generate
coherent and engaging content with arbitrary size and controllable style,
concept, and layout, all of which are essential for visual storytelling. To
overcome the shortcomings of previous methods including repetitive content,
style inconsistency, and lack of controllability, we propose MagicScroll, a
multi-layered, progressive diffusion-based image generation framework with a
novel semantic-aware denoising process. The model enables fine-grained control
over the generated image on object, scene, and background levels with text,
image, and layout conditions. We also establish the first benchmark for
nontypical aspect-ratio image generation for visual storytelling including
mediums like paintings, comics, and cinematic panoramas, with customized
metrics for systematic evaluation. Through comparative and ablation studies,
MagicScroll showcases promising results in aligning with the narrative text,
improving visual coherence, and engaging the audience. We plan to release the
code and benchmark in the hope of a better collaboration between AI researchers
and creative practitioners involving visual storytelling.
MagicScroll is a diffusion-based image generation framework that ensures coherence and style consistency across diverse aspect ratios for visual storytelling, leveraging text, image, and layout conditions.
diffusion-based image generation semantic-aware denoising process aspect-ratio images visual storytelling narrative text visual coherence