FineControlNet: Fine-level Text Control for Image Generation with
  Spatially Aligned Text Control Injection
Recently introduced ControlNet has the ability to steer the text-driven image
generation process with geometric input such as human 2D pose, or edge
features. While ControlNet provides control over the geometric form of the
instances in the generated image, it lacks the capability to dictate the visual
appearance of each instance. We present FineControlNet to provide fine control
over each instance's appearance while maintaining the precise pose control
capability. Specifically, we develop and demonstrate FineControlNet with
geometric control via human pose images and appearance control via
instance-level text prompts. The spatial alignment of instance-specific text
prompts and 2D poses in latent space enables the fine control capabilities of
FineControlNet. We evaluate the performance of FineControlNet with rigorous
comparison against state-of-the-art pose-conditioned text-to-image diffusion
models. FineControlNet achieves superior performance in generating images that
follow the user-provided instance-specific text prompts and poses compared with
existing methods. Project webpage:
https://samsunglabs.github.io/FineControlNet-project-page
657bd4163a230db5893b6217
Hongsuk Choi
admin_assigned
2023-12-15T13:21:55.669Z
63d6fc386f49aa8230681920
https://cdn-avatars.huggingface.co/v1/production/uploads/63d6fc386f49aa8230681920/TIDKMuCn5wq9Q97f4x43M.png
Hongsuk Choi
user
Hongsuk
657bd4163a230db5893b6218
Isaac Kasahara
657bd4163a230db5893b6219
Selim Engin
admin_assigned
2023-12-15T13:22:09.883Z
651b2e641a90782f9c9eb3b1
/avatars/e9bb88e5820c794026850be48c87b267.svg
Selim Engin
user
ksengin
657bd4163a230db5893b621a
Moritz Graule
admin_assigned
2023-12-15T13:22:16.282Z
64e61db399ffdf86b53328c9
/avatars/8c3f413f183ea5ff75b1850ce2850760.svg
Moritz Graule
user
graulem
657bd4163a230db5893b621b
Nikhil Chavan-Dafle
657bd4163a230db5893b621c
Volkan Isler
FineControlNet: Fine-level Text Control for Image Generation with
  Spatially Aligned Text Control Injection
Recently introduced ControlNet has the ability to steer the text-driven image
generation process with geometric input such as human 2D pose, or edge
features. While ControlNet provides control over the geometric form of the
instances in the generated image, it lacks the capability to dictate the visual
appearance of each instance. We present FineControlNet to provide fine control
over each instance's appearance while maintaining the precise pose control
capability. Specifically, we develop and demonstrate FineControlNet with
geometric control via human pose images and appearance control via
instance-level text prompts. The spatial alignment of instance-specific text
prompts and 2D poses in latent space enables the fine control capabilities of
FineControlNet. We evaluate the performance of FineControlNet with rigorous
comparison against state-of-the-art pose-conditioned text-to-image diffusion
models. FineControlNet achieves superior performance in generating images that
follow the user-provided instance-specific text prompts and poses compared with
existing methods. Project webpage:
https://samsunglabs.github.io/FineControlNet-project-page
FineControlNet enhances text-driven image generation by allowing fine control over instance appearance and maintaining pose control, outperforming existing methods in generating images that align with user-provided prompts and poses.
ControlNet text-driven image generation geometric input human 2D pose edge features appearance control instance-level text prompts latent space spatial alignment pose-conditioned text-to-image diffusion models