MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
In this paper, we present MoMA: an open-vocabulary, training-free
personalized image model that boasts flexible zero-shot capabilities. As
foundational text-to-image models rapidly evolve, the demand for robust
image-to-image translation grows. Addressing this need, MoMA specializes in
subject-driven personalized image generation. Utilizing an open-source,
Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as
both a feature extractor and a generator. This approach effectively synergizes
reference image and text prompt information to produce valuable image features,
facilitating an image diffusion model. To better leverage the generated
features, we further introduce a novel self-attention shortcut method that
efficiently transfers image features to an image diffusion model, improving the
resemblance of the target object in generated images. Remarkably, as a
tuning-free plug-and-play module, our model requires only a single reference
image and outperforms existing methods in generating images with high detail
fidelity, enhanced identity-preservation and prompt faithfulness. Our work is
open-source, thereby providing universal access to these advancements.
6614c9137de7824887701c20
Kunpeng Song
6614c9137de7824887701c21
Yizhe Zhu
6614c9137de7824887701c22
Bingchen Liu
claimed_verified
2024-11-15T09:47:50.306Z
62f9e1f90697d2242198c1db
https://cdn-avatars.huggingface.co/v1/production/uploads/1660543471146-noauth.jpeg
bingchen liu
user
codegeasslbc
6614c9137de7824887701c23
Qing Yan
6614c9137de7824887701c24
Ahmed Elgammal
6614c9137de7824887701c25
Xiao Yang
claimed_verified
2024-07-03T07:41:44.144Z
62b3d2490385d6048985b42b
/avatars/1c4e534d086d53ef1057fbc00334350a.svg
Xiao Yang
user
seravee008
MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation
In this paper, we present MoMA: an open-vocabulary, training-free
personalized image model that boasts flexible zero-shot capabilities. As
foundational text-to-image models rapidly evolve, the demand for robust
image-to-image translation grows. Addressing this need, MoMA specializes in
subject-driven personalized image generation. Utilizing an open-source,
Multimodal Large Language Model (MLLM), we train MoMA to serve a dual role as
both a feature extractor and a generator. This approach effectively synergizes
reference image and text prompt information to produce valuable image features,
facilitating an image diffusion model. To better leverage the generated
features, we further introduce a novel self-attention shortcut method that
efficiently transfers image features to an image diffusion model, improving the
resemblance of the target object in generated images. Remarkably, as a
tuning-free plug-and-play module, our model requires only a single reference
image and outperforms existing methods in generating images with high detail
fidelity, enhanced identity-preservation and prompt faithfulness. Our work is
open-source, thereby providing universal access to these advancements.
MoMA, an open-vocabulary and training-free model, uses a Multimodal Large Language Model to generate high-detail, identity-preserved images through an image diffusion model enhanced by a self-attention shortcut.
Multimodal Large Language Model feature extractor image diffusion model self-attention shortcut open-vocabulary zero-shot capabilities image-to-image translation reference image text prompt high detail fidelity identity-preservation prompt faithfulness