title: ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting
    with Diffusion Transformers
summary: Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the
  inherent in-context generation capabilities of pretrained diffusion
  transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks
  with minimal or no architectural modifications. These capabilities are unlocked
  by concatenating self-attention tokens across multiple input and target images,
  combined with grouped and masked generation pipelines. Building upon this
  foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive
  visual generation framework that leverages pretrained diffusion transformers in
  their original form, requiring no additional tuning, adapters, or
  modifications. Users can interact with ChatDiT to create interleaved text-image
  articles, multi-page picture books, edit images, design IP derivatives, or
  develop character design settings, all through free-form natural language
  across one or more conversational rounds. At its core, ChatDiT employs a
  multi-agent system comprising three key components: an Instruction-Parsing
  agent that interprets user-uploaded images and instructions, a
  Strategy-Planning agent that devises single-step or multi-step generation
  actions, and an Execution agent that performs these actions using an in-context
  toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench
  arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with
  diverse instructions and varying numbers of input and target images. Despite
  its simplicity and training-free approach, ChatDiT surpasses all competitors,
  including those specifically designed and trained on extensive multi-task
  datasets. We further identify key limitations of pretrained DiTs in zero-shot
  adapting to tasks. We release all code, agents, results, and intermediate
  outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
paper_authors._id: 67638e7f2ac5b74c7449ccc0
paper_authors.name: Lianghua Huang
paper_authors._id: 67638e7f2ac5b74c7449ccc1
paper_authors.name: Wei Wang
paper_authors._id: 67638e7f2ac5b74c7449ccc2
paper_authors.name: Zhi-Fan Wu
paper_authors._id: 67638e7f2ac5b74c7449ccc3
paper_authors.name: Yupeng Shi
paper_authors._id: 67638e7f2ac5b74c7449ccc4
paper_authors.name: Chen Liang
paper_authors._id: 67638e7f2ac5b74c7449ccc5
paper_authors.name: Tong Shen
paper_authors._id: 67638e7f2ac5b74c7449ccc6
paper_authors.name: Han Zhang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-12-19T03:14:37.161Z
paper_authors._id: 63c3b12da361002ba0ae5381
paper_authors.avatarUrl: /avatars/f505a612c7698b5975a1daa3cca6a1c3.svg
paper_authors.fullname: Han Zhang
paper_authors.type: user
paper_authors.user: bibona
paper_authors._id: 67638e7f2ac5b74c7449ccc7
paper_authors.name: Huanzhang Dou
paper_authors._id: 67638e7f2ac5b74c7449ccc8
paper_authors.name: Yu Liu
paper_authors._id: 67638e7f2ac5b74c7449ccc9
paper_authors.name: Jingren Zhou
paper_title: ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting
    with Diffusion Transformers
paper_summary: Recent research arXiv:2410.15027 arXiv:2410.23775 has highlighted the
  inherent in-context generation capabilities of pretrained diffusion
  transformers (DiTs), enabling them to seamlessly adapt to diverse visual tasks
  with minimal or no architectural modifications. These capabilities are unlocked
  by concatenating self-attention tokens across multiple input and target images,
  combined with grouped and masked generation pipelines. Building upon this
  foundation, we present ChatDiT, a zero-shot, general-purpose, and interactive
  visual generation framework that leverages pretrained diffusion transformers in
  their original form, requiring no additional tuning, adapters, or
  modifications. Users can interact with ChatDiT to create interleaved text-image
  articles, multi-page picture books, edit images, design IP derivatives, or
  develop character design settings, all through free-form natural language
  across one or more conversational rounds. At its core, ChatDiT employs a
  multi-agent system comprising three key components: an Instruction-Parsing
  agent that interprets user-uploaded images and instructions, a
  Strategy-Planning agent that devises single-step or multi-step generation
  actions, and an Execution agent that performs these actions using an in-context
  toolkit of diffusion transformers. We thoroughly evaluate ChatDiT on IDEA-Bench
  arXiv:2412.11767, comprising 100 real-world design tasks and 275 cases with
  diverse instructions and varying numbers of input and target images. Despite
  its simplicity and training-free approach, ChatDiT surpasses all competitors,
  including those specifically designed and trained on extensive multi-task
  datasets. We further identify key limitations of pretrained DiTs in zero-shot
  adapting to tasks. We release all code, agents, results, and intermediate
  outputs to facilitate further research at https://github.com/ali-vilab/ChatDiT
paper_ai_keywords: diffusion transformers (DiTs) self-attention tokens grouped and masked generation pipelines ChatDiT Instruction-Parsing agent Strategy-Planning agent Execution agent IDEA-Bench in-context toolkit