title: ROICtrl: Boosting Instance Control for Visual Generation
summary: Natural language often struggles to accurately associate positional and
  attribute information with multiple instances, which limits current text-based
  visual generation models to simpler compositions featuring only a few dominant
  instances. To address this limitation, this work enhances diffusion models by
  introducing regional instance control, where each instance is governed by a
  bounding box paired with a free-form caption. Previous methods in this area
  typically rely on implicit position encoding or explicit attention masks to
  separate regions of interest (ROIs), resulting in either inaccurate coordinate
  injection or large computational overhead. Inspired by ROI-Align in object
  detection, we introduce a complementary operation called ROI-Unpool. Together,
  ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI
  manipulation on high-resolution feature maps for visual generation. Building on
  ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that
  enables precise regional instance control. ROICtrl is compatible with
  community-finetuned diffusion models, as well as with existing spatial-based
  add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg,
  IP-Adapter, ED-LoRA), extending their applications to multi-instance
  generation. Experiments show that ROICtrl achieves superior performance in
  regional instance control while significantly reducing computational costs.
paper_authors._id: 6747cff9cf8e0c077f37d066
paper_authors.name: Yuchao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T09:54:59.262Z
paper_authors._id: 63021630a35b21bd8a53305a
paper_authors.avatarUrl: /avatars/7a7e8b39749eda61e57d8a1908726558.svg
paper_authors.fullname: Gu Yuchao
paper_authors.type: user
paper_authors.user: guyuchao
paper_authors._id: 6747cff9cf8e0c077f37d067
paper_authors.name: Yipin Zhou
paper_authors._id: 6747cff9cf8e0c077f37d068
paper_authors.name: Yunfan Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T09:55:39.519Z
paper_authors._id: 63b8c5ec3d5ccb207f57f03a
paper_authors.avatarUrl: /avatars/070f535e946b5cadb82f61e5ca0a90be.svg
paper_authors.fullname: Yunfan Ye
paper_authors.type: user
paper_authors.user: 365sleep
paper_authors._id: 6747cff9cf8e0c077f37d069
paper_authors.name: Yixin Nie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:47.282Z
paper_authors._id: 5f7919d5f1e7ef6e919a1f55
paper_authors.avatarUrl: /avatars/fb747fb43b65ccdfb08d7bf71525f92f.svg
paper_authors.fullname: Yixin Nie
paper_authors.type: user
paper_authors.user: ynie
paper_authors._id: 6747cff9cf8e0c077f37d06a
paper_authors.name: Licheng Yu
paper_authors._id: 6747cff9cf8e0c077f37d06b
paper_authors.name: Pingchuan Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:45.738Z
paper_authors._id: 64c163ecd9bdb0d64d58f716
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c163ecd9bdb0d64d58f716/ln8536t4XH4ZUYxdSmlId.png
paper_authors.fullname: Pingchuan Ma
paper_authors.type: user
paper_authors.user: pcma
paper_authors._id: 6747cff9cf8e0c077f37d06c
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T09:54:20.686Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 6747cff9cf8e0c077f37d06d
paper_authors.name: Mike Zheng Shou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T09:54:09.470Z
paper_authors._id: 661ab3da2b14565c7acccf5c
paper_authors.avatarUrl: /avatars/fa4fc03664803e02aede4d4c3d50b393.svg
paper_authors.fullname: Mike Zheng Shou
paper_authors.type: user
paper_authors.user: AnalMom
paper_title: ROICtrl: Boosting Instance Control for Visual Generation
paper_summary: Natural language often struggles to accurately associate positional and
  attribute information with multiple instances, which limits current text-based
  visual generation models to simpler compositions featuring only a few dominant
  instances. To address this limitation, this work enhances diffusion models by
  introducing regional instance control, where each instance is governed by a
  bounding box paired with a free-form caption. Previous methods in this area
  typically rely on implicit position encoding or explicit attention masks to
  separate regions of interest (ROIs), resulting in either inaccurate coordinate
  injection or large computational overhead. Inspired by ROI-Align in object
  detection, we introduce a complementary operation called ROI-Unpool. Together,
  ROI-Align and ROI-Unpool enable explicit, efficient, and accurate ROI
  manipulation on high-resolution feature maps for visual generation. Building on
  ROI-Unpool, we propose ROICtrl, an adapter for pretrained diffusion models that
  enables precise regional instance control. ROICtrl is compatible with
  community-finetuned diffusion models, as well as with existing spatial-based
  add-ons (\eg, ControlNet, T2I-Adapter) and embedding-based add-ons (\eg,
  IP-Adapter, ED-LoRA), extending their applications to multi-instance
  generation. Experiments show that ROICtrl achieves superior performance in
  regional instance control while significantly reducing computational costs.
paper_ai_summary: ROICtrl, an adapter for diffusion models using ROI-Align and ROI-Unpool, improves regional instance control in text-based visual generation with lower computational costs.
paper_ai_keywords: diffusion models regional instance control bounding box free-form caption regions of interest implicit position encoding explicit attention masks ROI-Align ROI-Unpool ROICtrl ControlNet T2I-Adapter IP-Adapter ED-LoRA multi-instance generation