IFAdapter: Instance Feature Control for Grounded Text-to-Image
  Generation
While Text-to-Image (T2I) diffusion models excel at generating visually
appealing images of individual instances, they struggle to accurately position
and control the features generation of multiple instances. The Layout-to-Image
(L2I) task was introduced to address the positioning challenges by
incorporating bounding boxes as spatial control signals, but it still falls
short in generating precise instance features. In response, we propose the
Instance Feature Generation (IFG) task, which aims to ensure both positional
accuracy and feature fidelity in generated instances. To address the IFG task,
we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances
feature depiction by incorporating additional appearance tokens and utilizing
an Instance Semantic Map to align instance-level features with spatial
locations. The IFAdapter guides the diffusion process as a plug-and-play
module, making it adaptable to various community models. For evaluation, we
contribute an IFG benchmark and develop a verification pipeline to objectively
compare models' abilities to generate instances with accurate positioning and
features. Experimental results demonstrate that IFAdapter outperforms other
models in both quantitative and qualitative evaluations.
66e3d6c2d3fa06c30282f5bd
Yinwei Wu
claimed_verified
2024-09-13T07:20:43.493Z
650e77418d01590937b71f1d
/avatars/9e2ea9c034c39fa6de59789e90919a21.svg
AWEI
user
WuYW
66e3d6c2d3fa06c30282f5be
Xianpan Zhou
admin_assigned
2024-09-13T08:08:04.274Z
649d2d6a4e03557284229376
/avatars/6b80dd6291e5d8fd89b10a574bc37558.svg
XianpanZhou
user
tinytigerpan
66e3d6c2d3fa06c30282f5bf
Bing Ma
66e3d6c2d3fa06c30282f5c0
Xuefeng Su
admin_assigned
2024-09-13T08:08:35.328Z
64abcf623c974392fbc5ebf6
/avatars/b8ea4ba990beefaebd42b3ff8807d458.svg
xuefeng
user
suxuefeng
66e3d6c2d3fa06c30282f5c1
Kai Ma
66e3d6c2d3fa06c30282f5c2
Xinchao Wang
admin_assigned
2024-09-13T08:09:07.498Z
63fc03a50aab060792ffef39
/avatars/9d5b1bb2a41928e08176b703935133ab.svg
Wangxinchao
user
wxcTest
IFAdapter: Instance Feature Control for Grounded Text-to-Image
  Generation
While Text-to-Image (T2I) diffusion models excel at generating visually
appealing images of individual instances, they struggle to accurately position
and control the features generation of multiple instances. The Layout-to-Image
(L2I) task was introduced to address the positioning challenges by
incorporating bounding boxes as spatial control signals, but it still falls
short in generating precise instance features. In response, we propose the
Instance Feature Generation (IFG) task, which aims to ensure both positional
accuracy and feature fidelity in generated instances. To address the IFG task,
we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances
feature depiction by incorporating additional appearance tokens and utilizing
an Instance Semantic Map to align instance-level features with spatial
locations. The IFAdapter guides the diffusion process as a plug-and-play
module, making it adaptable to various community models. For evaluation, we
contribute an IFG benchmark and develop a verification pipeline to objectively
compare models' abilities to generate instances with accurate positioning and
features. Experimental results demonstrate that IFAdapter outperforms other
models in both quantitative and qualitative evaluations.
The Instance Feature Adapter enhances Text-to-Image diffusion models by accurately positioning and depicting instance features using appearance tokens and semantic maps.
T2I diffusion models Layout-to-Image task Instance Feature Generation task Instance Feature Adapter appearance tokens Instance Semantic Map diffusion process plug-and-play module IFG benchmark verification pipeline