ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting
  with Diffusion Transformers
Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the
inherent in-context generation capabilities of pretrained diffusion
transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks
with minimal or no architectural modifications. These capabilities are unlocked
by concatenating self-attention tokens across multiple input and target images,
combined with grouped and masked generation pipelines. Building upon this
foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive
visual generation framework that leverages pretrained diffusion transformers in
their original form, requiring no additional tuning, adapters, or
modifications. Users can interact with ChatDiT to create interleaved text-image
articles, multi-page picture books, edit images, design IP derivatives, or
develop character design settings, all through free-form natural language
across one or more conversational rounds. At its core, ChatDiT employs a
multi-agent system comprising three key components: an Instruction-Parsing
agent that interprets user-uploaded images and instructions, a
Strategy-Planning agent that devises single-step or multi-step generation
actions, and an Execution agent that performs these actions using an in-context
toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench
arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with
diverse instructions and varying numbers of input and target images. Despite
its simplicity and training-free approach, ChatDiT surpasses all competitors,
including those specifically designed and trained on extensive multi-task
datasets. We further identify key limitations of pretrained DiTs in zero-shot
adapting to tasks. We release all code, agents, results, and intermediate
outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
67638e7f2ac5b74c7449ccc0
Lianghua Huang
67638e7f2ac5b74c7449ccc1
Wei Wang
67638e7f2ac5b74c7449ccc2
Zhi-Fan Wu
67638e7f2ac5b74c7449ccc3
Yupeng Shi
67638e7f2ac5b74c7449ccc4
Chen Liang
67638e7f2ac5b74c7449ccc5
Tong Shen
67638e7f2ac5b74c7449ccc6
Han Zhang
extracted_confirmed
2024-12-19T03:14:37.161Z
63c3b12da361002ba0ae5381
/avatars/f505a612c7698b5975a1daa3cca6a1c3.svg
Han Zhang
user
bibona
67638e7f2ac5b74c7449ccc7
Huanzhang Dou
67638e7f2ac5b74c7449ccc8
Yu Liu
67638e7f2ac5b74c7449ccc9
Jingren Zhou
ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting
  with Diffusion Transformers
Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the
inherent in-context generation capabilities of pretrained diffusion
transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks
with minimal or no architectural modifications. These capabilities are unlocked
by concatenating self-attention tokens across multiple input and target images,
combined with grouped and masked generation pipelines. Building upon this
foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive
visual generation framework that leverages pretrained diffusion transformers in
their original form, requiring no additional tuning, adapters, or
modifications. Users can interact with ChatDiT to create interleaved text-image
articles, multi-page picture books, edit images, design IP derivatives, or
develop character design settings, all through free-form natural language
across one or more conversational rounds. At its core, ChatDiT employs a
multi-agent system comprising three key components: an Instruction-Parsing
agent that interprets user-uploaded images and instructions, a
Strategy-Planning agent that devises single-step or multi-step generation
actions, and an Execution agent that performs these actions using an in-context
toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench
arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with
diverse instructions and varying numbers of input and target images. Despite
its simplicity and training-free approach, ChatDiT surpasses all competitors,
including those specifically designed and trained on extensive multi-task
datasets. We further identify key limitations of pretrained DiTs in zero-shot
adapting to tasks. We release all code, agents, results, and intermediate
outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
ChatDiT is a zero-shot, interactive visual generation framework using pretrained diffusion transformers without additional tuning, capable of creating diverse content from text and image inputs.
diffusion transformers DiTs self-attention tokens grouped generation masked generation Instruction-Parsing agent Strategy-Planning agent Execution agent IDEA-Bench