title: MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary
    Instance Segmentation
summary: We present MosaicFusion, a simple yet effective diffusion-based data
  augmentation approach for large vocabulary instance segmentation. Our method is
  training-free and does not rely on any label supervision. Two key designs
  enable us to employ an off-the-shelf text-to-image diffusion model as a useful
  dataset generator for object instances and mask annotations. First, we divide
  an image canvas into several regions and perform a single round of diffusion
  process to generate multiple instances simultaneously, conditioning on
  different text prompts. Second, we obtain corresponding instance masks by
  aggregating cross-attention maps associated with object prompts across layers
  and diffusion time steps, followed by simple thresholding and edge-aware
  refinement processing. Without bells and whistles, our MosaicFusion can produce
  a significant amount of synthetic labeled data for both rare and novel
  categories. Experimental results on the challenging LVIS long-tailed and
  open-vocabulary benchmarks demonstrate that MosaicFusion can significantly
  improve the performance of existing instance segmentation models, especially
  for rare and novel categories. Code will be released at
  https://github.com/Jiahao000/MosaicFusion.
paper_authors._id: 6511195a73c96e4a4045d298
paper_authors.name: Jiahao Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:00:53.097Z
paper_authors._id: 62fcf93cc1588e1d4c66453e
paper_authors.avatarUrl: /avatars/5ebf58aca34e7eb7cb56d1188118e7ed.svg
paper_authors.fullname: Jiahao Xie
paper_authors.type: user
paper_authors.user: Jiahao000
paper_authors._id: 6511195a73c96e4a4045d299
paper_authors.name: Wei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T13:02:09.004Z
paper_authors._id: 63bf9f26da08ed0544007446
paper_authors.avatarUrl: /avatars/836e857e2eb63fee202e6a5aaa0f9faa.svg
paper_authors.fullname: Wei Li
paper_authors.type: user
paper_authors.user: weivision
paper_authors._id: 6511195a73c96e4a4045d29a
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:02:02.437Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 6511195a73c96e4a4045d29b
paper_authors.name: Ziwei Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-25T11:02:13.967Z
paper_authors._id: 62ab1ac1d48b4d8b048a3473
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
paper_authors.fullname: Ziwei Liu
paper_authors.type: user
paper_authors.user: liuziwei7
paper_authors._id: 6511195a73c96e4a4045d29c
paper_authors.name: Yew Soon Ong
paper_authors._id: 6511195a73c96e4a4045d29d
paper_authors.name: Chen Change Loy
paper_title: MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary
    Instance Segmentation
paper_summary: We present MosaicFusion, a simple yet effective diffusion-based data
  augmentation approach for large vocabulary instance segmentation. Our method is
  training-free and does not rely on any label supervision. Two key designs
  enable us to employ an off-the-shelf text-to-image diffusion model as a useful
  dataset generator for object instances and mask annotations. First, we divide
  an image canvas into several regions and perform a single round of diffusion
  process to generate multiple instances simultaneously, conditioning on
  different text prompts. Second, we obtain corresponding instance masks by
  aggregating cross-attention maps associated with object prompts across layers
  and diffusion time steps, followed by simple thresholding and edge-aware
  refinement processing. Without bells and whistles, our MosaicFusion can produce
  a significant amount of synthetic labeled data for both rare and novel
  categories. Experimental results on the challenging LVIS long-tailed and
  open-vocabulary benchmarks demonstrate that MosaicFusion can significantly
  improve the performance of existing instance segmentation models, especially
  for rare and novel categories. Code will be released at
  https://github.com/Jiahao000/MosaicFusion.
paper_ai_summary: MosaicFusion improves instance segmentation by generating synthetic labeled data through a diffusion-based approach using text-to-image models and cross-attention maps.
paper_ai_keywords: diffusion-based data augmentation instance segmentation text-to-image diffusion model cross-attention maps LVIS long-tailed open-vocabulary