title: FineControlNet: Fine-level Text Control for Image Generation with
    Spatially Aligned Text Control Injection
summary: Recently introduced ControlNet has the ability to steer the text-driven image
  generation process with geometric input such as human 2D pose, or edge
  features. While ControlNet provides control over the geometric form of the
  instances in the generated image, it lacks the capability to dictate the visual
  appearance of each instance. We present FineControlNet to provide fine control
  over each instance's appearance while maintaining the precise pose control
  capability. Specifically, we develop and demonstrate FineControlNet with
  geometric control via human pose images and appearance control via
  instance-level text prompts. The spatial alignment of instance-specific text
  prompts and 2D poses in latent space enables the fine control capabilities of
  FineControlNet. We evaluate the performance of FineControlNet with rigorous
  comparison against state-of-the-art pose-conditioned text-to-image diffusion
  models. FineControlNet achieves superior performance in generating images that
  follow the user-provided instance-specific text prompts and poses compared with
  existing methods. Project webpage:
  https://samsunglabs.github.io/FineControlNet-project-page
paper_authors._id: 657bd4163a230db5893b6217
paper_authors.name: Hongsuk Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:21:55.669Z
paper_authors._id: 63d6fc386f49aa8230681920
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63d6fc386f49aa8230681920/TIDKMuCn5wq9Q97f4x43M.png
paper_authors.fullname: Hongsuk Choi
paper_authors.type: user
paper_authors.user: Hongsuk
paper_authors._id: 657bd4163a230db5893b6218
paper_authors.name: Isaac Kasahara
paper_authors._id: 657bd4163a230db5893b6219
paper_authors.name: Selim Engin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:22:09.883Z
paper_authors._id: 651b2e641a90782f9c9eb3b1
paper_authors.avatarUrl: /avatars/e9bb88e5820c794026850be48c87b267.svg
paper_authors.fullname: Selim Engin
paper_authors.type: user
paper_authors.user: ksengin
paper_authors._id: 657bd4163a230db5893b621a
paper_authors.name: Moritz Graule
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:22:16.282Z
paper_authors._id: 64e61db399ffdf86b53328c9
paper_authors.avatarUrl: /avatars/8c3f413f183ea5ff75b1850ce2850760.svg
paper_authors.fullname: Moritz Graule
paper_authors.type: user
paper_authors.user: graulem
paper_authors._id: 657bd4163a230db5893b621b
paper_authors.name: Nikhil Chavan-Dafle
paper_authors._id: 657bd4163a230db5893b621c
paper_authors.name: Volkan Isler
paper_title: FineControlNet: Fine-level Text Control for Image Generation with
    Spatially Aligned Text Control Injection
paper_summary: Recently introduced ControlNet has the ability to steer the text-driven image
  generation process with geometric input such as human 2D pose, or edge
  features. While ControlNet provides control over the geometric form of the
  instances in the generated image, it lacks the capability to dictate the visual
  appearance of each instance. We present FineControlNet to provide fine control
  over each instance's appearance while maintaining the precise pose control
  capability. Specifically, we develop and demonstrate FineControlNet with
  geometric control via human pose images and appearance control via
  instance-level text prompts. The spatial alignment of instance-specific text
  prompts and 2D poses in latent space enables the fine control capabilities of
  FineControlNet. We evaluate the performance of FineControlNet with rigorous
  comparison against state-of-the-art pose-conditioned text-to-image diffusion
  models. FineControlNet achieves superior performance in generating images that
  follow the user-provided instance-specific text prompts and poses compared with
  existing methods. Project webpage:
  https://samsunglabs.github.io/FineControlNet-project-page
paper_ai_keywords: ControlNet text-driven image generation geometric input human 2D pose edge features appearance control instance-level text prompts latent space spatial alignment FineControlNet pose-conditioned text-to-image diffusion models