StoryMaker: Towards Holistic Consistent Characters in Text-to-image
  Generation
Tuning-free personalized image generation methods have achieved significant
success in maintaining facial consistency, i.e., identities, even with multiple
characters. However, the lack of holistic consistency in scenes with multiple
characters hampers these methods' ability to create a cohesive narrative. In
this paper, we introduce StoryMaker, a personalization solution that preserves
not only facial consistency but also clothing, hairstyles, and body
consistency, thus facilitating the creation of a story through a series of
images. StoryMaker incorporates conditions based on face identities and cropped
character images, which include clothing, hairstyles, and bodies. Specifically,
we integrate the facial identity information with the cropped character images
using the Positional-aware Perceiver Resampler (PPR) to obtain distinct
character features. To prevent intermingling of multiple characters and the
background, we separately constrain the cross-attention impact regions of
different characters and the background using MSE loss with segmentation masks.
Additionally, we train the generation network conditioned on poses to promote
decoupling from poses. A LoRA is also employed to enhance fidelity and quality.
Experiments underscore the effectiveness of our approach. StoryMaker supports
numerous applications and is compatible with other societal plug-ins. Our
source codes and model weights are available at
https://github.com/RedAIGC/StoryMaker.
66ecd9e0fcc1c455f812a695
Zhengguang Zhou
admin_assigned
2024-09-20T08:55:24.898Z
64b51874849c9e54e0d55f2d
/avatars/07228522b73d591f40f4cd3794851752.svg
zhouzhengguang
user
huaquan
66ecd9e0fcc1c455f812a696
Jing Li
admin_assigned
2024-09-20T08:55:31.419Z
632c2a556bcb864974cc1558
/avatars/000cd418f438a2754366b94dcf0aee10.svg
Jing Li
user
jingli
66ecd9e0fcc1c455f812a697
Huaxia Li
admin_assigned
2024-09-20T08:55:37.541Z
64b894cbe436bbca16627f4b
/avatars/955908f51b2fbee9db8dd071d3f23d3c.svg
huaxiali
user
huaxiali
66ecd9e0fcc1c455f812a698
Nemo Chen
admin_assigned
2024-09-20T08:55:43.096Z
64c87825cbcae9f106cf90b0
https://cdn-avatars.huggingface.co/v1/production/uploads/64c87825cbcae9f106cf90b0/LAug686nylBUYojPLrkyb.png
Nemo CHEN
user
chenyibo89
66ecd9e0fcc1c455f812a699
Xu Tang
StoryMaker: Towards Holistic Consistent Characters in Text-to-image
  Generation
Tuning-free personalized image generation methods have achieved significant
success in maintaining facial consistency, i.e., identities, even with multiple
characters. However, the lack of holistic consistency in scenes with multiple
characters hampers these methods' ability to create a cohesive narrative. In
this paper, we introduce StoryMaker, a personalization solution that preserves
not only facial consistency but also clothing, hairstyles, and body
consistency, thus facilitating the creation of a story through a series of
images. StoryMaker incorporates conditions based on face identities and cropped
character images, which include clothing, hairstyles, and bodies. Specifically,
we integrate the facial identity information with the cropped character images
using the Positional-aware Perceiver Resampler (PPR) to obtain distinct
character features. To prevent intermingling of multiple characters and the
background, we separately constrain the cross-attention impact regions of
different characters and the background using MSE loss with segmentation masks.
Additionally, we train the generation network conditioned on poses to promote
decoupling from poses. A LoRA is also employed to enhance fidelity and quality.
Experiments underscore the effectiveness of our approach. StoryMaker supports
numerous applications and is compatible with other societal plug-ins. Our
source codes and model weights are available at
https://github.com/RedAIGC/StoryMaker.
StoryMaker enhances personalized image generation by maintaining holistic consistency of multiple characters in scenes, using a Positional-aware Perceiver Resampler and segment-specific cross-attention constraints.
Positional-aware Perceiver Resampler cross-attention segmentation masks MSE loss LoRA pose conditioning