PartCraft: Crafting Creative Objects by Parts
This paper propels creative control in generative visual AI by allowing users
to "select". Departing from traditional text or sketch-based methods, we for
the first time allow users to choose visual concepts by parts for their
creative endeavors. The outcome is fine-grained generation that precisely
captures selected visual concepts, ensuring a holistically faithful and
plausible result. To achieve this, we first parse objects into parts through
unsupervised feature clustering. Then, we encode parts into text tokens and
introduce an entropy-based normalized attention loss that operates on them.
This loss design enables our model to learn generic prior topology knowledge
about object's part composition, and further generalize to novel part
compositions to ensure the generation looks holistically faithful. Lastly, we
employ a bottleneck encoder to project the part tokens. This not only enhances
fidelity but also accelerates learning, by leveraging shared knowledge and
facilitating information exchange among instances. Visual results in the paper
and supplementary material showcase the compelling power of PartCraft in
crafting highly customized, innovative creations, exemplified by the "charming"
and creative birds. Code is released at https://github.com/kamwoh/partcraft.
668bec5cb37c6ac0b09d9657
Kam Woh Ng
admin_assigned
2024-07-09T10:35:36.484Z
646f9a9d5b8225e31ae5a408
/avatars/b6210993b51bb650ea8aa8c04e5fc351.svg
kamwoh
user
kamwoh
668bec5cb37c6ac0b09d9658
Xiatian Zhu
admin_assigned
2024-07-09T10:35:42.899Z
647ef9e7aa8c04bbf9362a8a
/avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
Xiatian Zhu
user
Xiatian-Zhu
668bec5cb37c6ac0b09d9659
Yi-Zhe Song
admin_assigned
2024-07-09T10:35:56.664Z
6624bd6f9999cbbc78460744
/avatars/00264325fdf277e290aac5a1206ba73f.svg
YIZHE SONG
user
JackeySong
668bec5cb37c6ac0b09d965a
Tao Xiang
PartCraft: Crafting Creative Objects by Parts
This paper propels creative control in generative visual AI by allowing users
to "select". Departing from traditional text or sketch-based methods, we for
the first time allow users to choose visual concepts by parts for their
creative endeavors. The outcome is fine-grained generation that precisely
captures selected visual concepts, ensuring a holistically faithful and
plausible result. To achieve this, we first parse objects into parts through
unsupervised feature clustering. Then, we encode parts into text tokens and
introduce an entropy-based normalized attention loss that operates on them.
This loss design enables our model to learn generic prior topology knowledge
about object's part composition, and further generalize to novel part
compositions to ensure the generation looks holistically faithful. Lastly, we
employ a bottleneck encoder to project the part tokens. This not only enhances
fidelity but also accelerates learning, by leveraging shared knowledge and
facilitating information exchange among instances. Visual results in the paper
and supplementary material showcase the compelling power of PartCraft in
crafting highly customized, innovative creations, exemplified by the "charming"
and creative birds. Code is released at https://github.com/kamwoh/partcraft.
PartCraft allows users to select visual concepts by parts, enabling fine-grained generation with high fidelity by parsing objects, encoding parts into text tokens, and using an entropy-based normalized attention loss and bottleneck encoder.
unsupervised feature clustering entropy-based normalized attention loss bottleneck encoder