title: ACE: All-round Creator and Editor Following Instructions via Diffusion
    Transformer
summary: Diffusion models have emerged as a powerful generative technology and have
  been found to be applicable in various scenarios. Most existing foundational
  diffusion models are primarily designed for text-guided visual generation and
  do not support multi-modal conditions, which are essential for many visual
  editing tasks. This limitation prevents these foundational diffusion models
  from serving as a unified model in the field of visual generation, like GPT-4
  in the natural language processing field. In this work, we propose ACE, an
  All-round Creator and Editor, which achieves comparable performance compared to
  those expert models in a wide range of visual generation tasks. To achieve this
  goal, we first introduce a unified condition format termed Long-context
  Condition Unit (LCU), and propose a novel Transformer-based diffusion model
  that uses LCU as input, aiming for joint training across various generation and
  editing tasks. Furthermore, we propose an efficient data collection approach to
  address the issue of the absence of available training data. It involves
  acquiring pairwise images with synthesis-based or clustering-based pipelines
  and supplying these pairs with accurate textual instructions by leveraging a
  fine-tuned multi-modal large language model. To comprehensively evaluate the
  performance of our model, we establish a benchmark of manually annotated pairs
  data across a variety of visual generation tasks. The extensive experimental
  results demonstrate the superiority of our model in visual generation fields.
  Thanks to the all-in-one capabilities of our model, we can easily build a
  multi-modal chat system that responds to any interactive request for image
  creation using a single model to serve as the backend, avoiding the cumbersome
  pipeline typically employed in visual agents. Code and models will be available
  on the project page: https://ali-vilab.github.io/ace-page/.
paper_authors._id: 66fcb2fab3f1da2ed11a5be6
paper_authors.name: Zhen Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T16:09:31.178Z
paper_authors._id: 647ffabf28b737d7b9462eb2
paper_authors.avatarUrl: /avatars/210441fc6645d08b36ad43734108f914.svg
paper_authors.fullname: Zhen Han
paper_authors.type: user
paper_authors.user: hanzhn
paper_authors._id: 66fcb2fab3f1da2ed11a5be7
paper_authors.name: Zeyinzi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:07:09.512Z
paper_authors._id: 643d278b482011f5f2bd0fae
paper_authors.avatarUrl: /avatars/70b8a7ffbfa2a1c4b6f5ff5e2b96b7bf.svg
paper_authors.fullname: jiangzeyinzi
paper_authors.type: user
paper_authors.user: jiangzeyinzi
paper_authors._id: 66fcb2fab3f1da2ed11a5be8
paper_authors.name: Yulin Pan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-08T07:19:36.366Z
paper_authors._id: 65a9e000cab1bfcc3af8cfe2
paper_authors.avatarUrl: /avatars/ddd0062dbeb5eb976841cec8a63c64a6.svg
paper_authors.fullname: Yulin Pan
paper_authors.type: user
paper_authors.user: pan-yl
paper_authors._id: 66fcb2fab3f1da2ed11a5be9
paper_authors.name: Jingfeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:06:31.815Z
paper_authors._id: 66f0e0262aee3cb7e981bbac
paper_authors.avatarUrl: /avatars/f8f1e70469b5e047dc6e0e9dec6c5bc1.svg
paper_authors.fullname: Jingfeng Zhang
paper_authors.type: user
paper_authors.user: jingfengzhang
paper_authors._id: 66fcb2fab3f1da2ed11a5bea
paper_authors.name: Chaojie Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:06:26.022Z
paper_authors._id: 6458970cab9a44f42f620a80
paper_authors.avatarUrl: /avatars/f9779b0621c931f922440fec95342444.svg
paper_authors.fullname: chaojie mao
paper_authors.type: user
paper_authors.user: chaojiemao
paper_authors._id: 66fcb2fab3f1da2ed11a5beb
paper_authors.name: Chenwei Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:06:19.596Z
paper_authors._id: 66592c72f4124d863fd55574
paper_authors.avatarUrl: /avatars/98f0d5e6ba3728e8a1164aa5188a3298.svg
paper_authors.fullname: Chenwei Xie
paper_authors.type: user
paper_authors.user: chenweix7
paper_authors._id: 66fcb2fab3f1da2ed11a5bec
paper_authors.name: Yu Liu
paper_authors._id: 66fcb2fab3f1da2ed11a5bed
paper_authors.name: Jingren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-02T09:05:23.903Z
paper_authors._id: 602f88f5e8149a962412a667
paper_authors.avatarUrl: /avatars/b78f0e583df8e5d5e3365934fe5f4900.svg
paper_authors.fullname: Zhou
paper_authors.type: user
paper_authors.user: Jingren
paper_title: ACE: All-round Creator and Editor Following Instructions via Diffusion
    Transformer
paper_summary: Diffusion models have emerged as a powerful generative technology and have
  been found to be applicable in various scenarios. Most existing foundational
  diffusion models are primarily designed for text-guided visual generation and
  do not support multi-modal conditions, which are essential for many visual
  editing tasks. This limitation prevents these foundational diffusion models
  from serving as a unified model in the field of visual generation, like GPT-4
  in the natural language processing field. In this work, we propose ACE, an
  All-round Creator and Editor, which achieves comparable performance compared to
  those expert models in a wide range of visual generation tasks. To achieve this
  goal, we first introduce a unified condition format termed Long-context
  Condition Unit (LCU), and propose a novel Transformer-based diffusion model
  that uses LCU as input, aiming for joint training across various generation and
  editing tasks. Furthermore, we propose an efficient data collection approach to
  address the issue of the absence of available training data. It involves
  acquiring pairwise images with synthesis-based or clustering-based pipelines
  and supplying these pairs with accurate textual instructions by leveraging a
  fine-tuned multi-modal large language model. To comprehensively evaluate the
  performance of our model, we establish a benchmark of manually annotated pairs
  data across a variety of visual generation tasks. The extensive experimental
  results demonstrate the superiority of our model in visual generation fields.
  Thanks to the all-in-one capabilities of our model, we can easily build a
  multi-modal chat system that responds to any interactive request for image
  creation using a single model to serve as the backend, avoiding the cumbersome
  pipeline typically employed in visual agents. Code and models will be available
  on the project page: https://ali-vilab.github.io/ace-page/.
paper_ai_summary: ACE, a unified Transformer-based diffusion model, achieves comparable performance to expert models across various visual generation tasks by using a Long-context Condition Unit and addressing data scarcity with a novel data collection method.
paper_ai_keywords: diffusion models Transformer-based diffusion model Long-context Condition Unit multi-modal conditions visual generation multi-modal chat system