title: BrushEdit: All-In-One Image Inpainting and Editing
summary: Image editing has advanced significantly with the development of diffusion
  models using both inversion-based and instruction-based methods. However,
  current inversion-based approaches struggle with big modifications (e.g.,
  adding or removing objects) due to the structured nature of inversion noise,
  which hinders substantial changes. Meanwhile, instruction-based methods often
  constrain users to black-box operations, limiting direct interaction for
  specifying editing regions and intensity. To address these limitations, we
  propose BrushEdit, a novel inpainting-based instruction-guided image editing
  paradigm, which leverages multimodal large language models (MLLMs) and image
  inpainting models to enable autonomous, user-friendly, and interactive
  free-form instruction editing. Specifically, we devise a system enabling
  free-form instruction editing by integrating MLLMs and a dual-branch image
  inpainting model in an agent-cooperative framework to perform editing category
  classification, main object identification, mask acquisition, and editing area
  inpainting. Extensive experiments show that our framework effectively combines
  MLLMs and inpainting models, achieving superior performance across seven
  metrics including mask region preservation and editing effect coherence.
paper_authors._id: 676044eedd3e9eeed65d9a23
paper_authors.name: Yaowei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:41:12.208Z
paper_authors._id: 6362801380c1a705a6ea54ac
paper_authors.avatarUrl: /avatars/041ad5abf9be42e336938f51ebb8746c.svg
paper_authors.fullname: Yaowei Li
paper_authors.type: user
paper_authors.user: Yw22
paper_authors._id: 676044eedd3e9eeed65d9a24
paper_authors.name: Yuxuan Bian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:41:18.245Z
paper_authors._id: 650447dd52ca06fef957f05d
paper_authors.avatarUrl: /avatars/511c11ac9b3cc7a162bda5e07f6ee0a3.svg
paper_authors.fullname: Yuxuan BIAN
paper_authors.type: user
paper_authors.user: BianYx
paper_authors._id: 676044eedd3e9eeed65d9a25
paper_authors.name: Xuan Ju
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:41:23.731Z
paper_authors._id: 62d4577bc85b0fcf7fde39bb
paper_authors.avatarUrl: /avatars/a3a5729e33ae89ce9ba408830db3c835.svg
paper_authors.fullname: Xuan Ju
paper_authors.type: user
paper_authors.user: juxuan27
paper_authors._id: 676044eedd3e9eeed65d9a26
paper_authors.name: Zhaoyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:08.717Z
paper_authors._id: 658409ceca19ccf6d9989add
paper_authors.avatarUrl: /avatars/3ac1dbd25e52435185babdeb3da28875.svg
paper_authors.fullname: Zhaoyang Zhang
paper_authors.type: user
paper_authors.user: ZyZcuhk
paper_authors._id: 676044eedd3e9eeed65d9a27
paper_authors.name: Ying Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-17T09:41:29.223Z
paper_authors._id: 63ca3ddc04c979828310bfcb
paper_authors.avatarUrl: /avatars/615e0d8622950b4408b40d550f02a894.svg
paper_authors.fullname: Ying Shan
paper_authors.type: user
paper_authors.user: yshan2u
paper_authors._id: 676044eedd3e9eeed65d9a28
paper_authors.name: Qiang Xu
paper_title: BrushEdit: All-In-One Image Inpainting and Editing
paper_summary: Image editing has advanced significantly with the development of diffusion
  models using both inversion-based and instruction-based methods. However,
  current inversion-based approaches struggle with big modifications (e.g.,
  adding or removing objects) due to the structured nature of inversion noise,
  which hinders substantial changes. Meanwhile, instruction-based methods often
  constrain users to black-box operations, limiting direct interaction for
  specifying editing regions and intensity. To address these limitations, we
  propose BrushEdit, a novel inpainting-based instruction-guided image editing
  paradigm, which leverages multimodal large language models (MLLMs) and image
  inpainting models to enable autonomous, user-friendly, and interactive
  free-form instruction editing. Specifically, we devise a system enabling
  free-form instruction editing by integrating MLLMs and a dual-branch image
  inpainting model in an agent-cooperative framework to perform editing category
  classification, main object identification, mask acquisition, and editing area
  inpainting. Extensive experiments show that our framework effectively combines
  MLLMs and inpainting models, achieving superior performance across seven
  metrics including mask region preservation and editing effect coherence.
paper_ai_keywords: diffusion models inversion-based methods instruction-based methods inversion noise BrushEdit inpainting-based instruction-guided image editing multimodal large language models (MLLMs) dual-branch image inpainting model agent-cooperative framework editing category classification main object identification mask acquisition editing area inpainting mask region preservation editing effect coherence