Piece it Together: Part-Based Concepting with IP-Priors
Advanced generative models excel at synthesizing images but often rely on
text-based conditioning. Visual designers, however, often work beyond language,
directly drawing inspiration from existing visual elements. In many cases,
these elements represent only fragments of a potential concept-such as an
uniquely structured wing, or a specific hairstyle-serving as inspiration for
the artist to explore how they can come together creatively into a coherent
whole. Recognizing this need, we introduce a generative framework that
seamlessly integrates a partial set of user-provided visual components into a
coherent composition while simultaneously sampling the missing parts needed to
generate a plausible and complete concept. Our approach builds on a strong and
underexplored representation space, extracted from IP-Adapter+, on which we
train IP-Prior, a lightweight flow-matching model that synthesizes coherent
compositions based on domain-specific priors, enabling diverse and
context-aware generations. Additionally, we present a LoRA-based fine-tuning
strategy that significantly improves prompt adherence in IP-Adapter+ for a
given task, addressing its common trade-off between reconstruction quality and
prompt adherence.
67d44cb6bb362707e0c548b9
Elad Richardson
67d44cb6bb362707e0c548ba
Kfir Goldberg
claimed_verified
2025-03-16T21:13:52.604Z
6381e8c72a85b6714b7eb5f7
https://cdn-avatars.huggingface.co/v1/production/uploads/6381e8c72a85b6714b7eb5f7/GTqChXYVI1Q11btXn1X36.jpeg
kfir goldberg
user
kfirgold99
67d44cb6bb362707e0c548bb
Yuval Alaluf
67d44cb6bb362707e0c548bc
Daniel Cohen-Or
Piece it Together: Part-Based Concepting with IP-Priors
Advanced generative models excel at synthesizing images but often rely on
text-based conditioning. Visual designers, however, often work beyond language,
directly drawing inspiration from existing visual elements. In many cases,
these elements represent only fragments of a potential concept-such as an
uniquely structured wing, or a specific hairstyle-serving as inspiration for
the artist to explore how they can come together creatively into a coherent
whole. Recognizing this need, we introduce a generative framework that
seamlessly integrates a partial set of user-provided visual components into a
coherent composition while simultaneously sampling the missing parts needed to
generate a plausible and complete concept. Our approach builds on a strong and
underexplored representation space, extracted from IP-Adapter+, on which we
train IP-Prior, a lightweight flow-matching model that synthesizes coherent
compositions based on domain-specific priors, enabling diverse and
context-aware generations. Additionally, we present a LoRA-based fine-tuning
strategy that significantly improves prompt adherence in IP-Adapter+ for a
given task, addressing its common trade-off between reconstruction quality and
prompt adherence.
A generative framework integrates user-provided visual components using a flow-matching model and LoRA-based fine-tuning to create coherent and context-aware compositions.
generative framework visual components flow-matching model IP-Prior IP-Adapter+ LoRA-based fine-tuning prompt adherence domain-specific priors