title: Jodi: Unification of Visual Generation and Understanding via Joint
    Modeling
summary: Visual generation and understanding are two deeply interconnected aspects of
  human intelligence, yet they have been traditionally treated as separate tasks
  in machine learning. In this paper, we propose Jodi, a diffusion framework that
  unifies visual generation and understanding by jointly modeling the image
  domain and multiple label domains. Specifically, Jodi is built upon a linear
  diffusion transformer along with a role switch mechanism, which enables it to
  perform three particular types of tasks: (1) joint generation, where the model
  simultaneously generates images and multiple labels; (2) controllable
  generation, where images are generated conditioned on any combination of
  labels; and (3) image perception, where multiple labels can be predicted at
  once from a given image. Furthermore, we present the Joint-1.6M dataset, which
  contains 200,000 high-quality images collected from public sources, automatic
  labels for 7 visual domains, and LLM-generated captions. Extensive experiments
  demonstrate that Jodi excels in both generation and understanding tasks and
  exhibits strong extensibility to a wider range of visual domains. Code is
  available at https://github.com/VIPL-GENUN/Jodi.
paper_authors._id: 6835334e0c0aff775f3eb6e2
paper_authors.name: Yifeng Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T08:15:08.170Z
paper_authors._id: 640c64779e5247967ff1e0b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678533946170-640c64779e5247967ff1e0b2.jpeg
paper_authors.fullname: Yifeng Xu
paper_authors.type: user
paper_authors.user: xyfJASON
paper_authors._id: 6835334e0c0aff775f3eb6e3
paper_authors.name: Zhenliang He
paper_authors._id: 6835334e0c0aff775f3eb6e4
paper_authors.name: Meina Kan
paper_authors._id: 6835334e0c0aff775f3eb6e5
paper_authors.name: Shiguang Shan
paper_authors._id: 6835334e0c0aff775f3eb6e6
paper_authors.name: Xilin Chen
paper_title: Jodi: Unification of Visual Generation and Understanding via Joint
    Modeling
paper_summary: Visual generation and understanding are two deeply interconnected aspects of
  human intelligence, yet they have been traditionally treated as separate tasks
  in machine learning. In this paper, we propose Jodi, a diffusion framework that
  unifies visual generation and understanding by jointly modeling the image
  domain and multiple label domains. Specifically, Jodi is built upon a linear
  diffusion transformer along with a role switch mechanism, which enables it to
  perform three particular types of tasks: (1) joint generation, where the model
  simultaneously generates images and multiple labels; (2) controllable
  generation, where images are generated conditioned on any combination of
  labels; and (3) image perception, where multiple labels can be predicted at
  once from a given image. Furthermore, we present the Joint-1.6M dataset, which
  contains 200,000 high-quality images collected from public sources, automatic
  labels for 7 visual domains, and LLM-generated captions. Extensive experiments
  demonstrate that Jodi excels in both generation and understanding tasks and
  exhibits strong extensibility to a wider range of visual domains. Code is
  available at https://github.com/VIPL-GENUN/Jodi.
paper_ai_summary: Jodi, a diffusion framework using a linear diffusion transformer and role switch mechanism, unifies visual generation and understanding, performing joint, controllable, and perceptual tasks effectively across multiple visual domains.
paper_ai_keywords: diffusion framework linear diffusion transformer role switch mechanism joint generation controllable generation image perception Joint-1.6M dataset visual domains LLM-generated captions