Jodi: Unification of Visual Generation and Understanding via Joint
  Modeling
Visual generation and understanding are two deeply interconnected aspects of
human intelligence, yet they have been traditionally treated as separate tasks
in machine learning. In this paper, we propose Jodi, a diffusion framework that
unifies visual generation and understanding by jointly modeling the image
domain and multiple label domains. Specifically, Jodi is built upon a linear
diffusion transformer along with a role switch mechanism, which enables it to
perform three particular types of tasks: (1) joint generation, where the model
simultaneously generates images and multiple labels; (2) controllable
generation, where images are generated conditioned on any combination of
labels; and (3) image perception, where multiple labels can be predicted at
once from a given image. Furthermore, we present the Joint-1.6M dataset, which
contains 200,000 high-quality images collected from public sources, automatic
labels for 7 visual domains, and LLM-generated captions. Extensive experiments
demonstrate that Jodi excels in both generation and understanding tasks and
exhibits strong extensibility to a wider range of visual domains. Code is
available at https://github.com/VIPL-GENUN/Jodi.
6835334e0c0aff775f3eb6e2
Yifeng Xu
claimed_verified
2025-05-27T08:15:08.170Z
640c64779e5247967ff1e0b2
https://cdn-avatars.huggingface.co/v1/production/uploads/1678533946170-640c64779e5247967ff1e0b2.jpeg
Yifeng Xu
user
xyfJASON
6835334e0c0aff775f3eb6e3
Zhenliang He
admin_assigned
2025-06-27T17:41:25.834Z
641b1ec5f902cc42730c75bf
https://cdn-avatars.huggingface.co/v1/production/uploads/641b1ec5f902cc42730c75bf/AxdaDo3fNWCq2UTWZy8bo.png
Lynn Ho
user
LynnHoHZL
6835334e0c0aff775f3eb6e4
Meina Kan
6835334e0c0aff775f3eb6e5
Shiguang Shan
6835334e0c0aff775f3eb6e6
Xilin Chen
Jodi: Unification of Visual Generation and Understanding via Joint
  Modeling
Visual generation and understanding are two deeply interconnected aspects of
human intelligence, yet they have been traditionally treated as separate tasks
in machine learning. In this paper, we propose Jodi, a diffusion framework that
unifies visual generation and understanding by jointly modeling the image
domain and multiple label domains. Specifically, Jodi is built upon a linear
diffusion transformer along with a role switch mechanism, which enables it to
perform three particular types of tasks: (1) joint generation, where the model
simultaneously generates images and multiple labels; (2) controllable
generation, where images are generated conditioned on any combination of
labels; and (3) image perception, where multiple labels can be predicted at
once from a given image. Furthermore, we present the Joint-1.6M dataset, which
contains 200,000 high-quality images collected from public sources, automatic
labels for 7 visual domains, and LLM-generated captions. Extensive experiments
demonstrate that Jodi excels in both generation and understanding tasks and
exhibits strong extensibility to a wider range of visual domains. Code is
available at https://github.com/VIPL-GENUN/Jodi.
Jodi, a diffusion framework using a linear diffusion transformer and role switch mechanism, unifies visual generation and understanding, performing joint, controllable, and perceptual tasks effectively across multiple visual domains.
diffusion framework linear diffusion transformer role switch mechanism joint generation controllable generation image perception Joint-1.6M dataset visual domains LLM-generated captions