UniFork: Exploring Modality Alignment for Unified Multimodal
  Understanding and Generation
Unified image understanding and generation has emerged as a promising
paradigm in multimodal artificial intelligence. Despite recent progress, the
optimal architectural design for such unified models remains an open challenge.
In this work, we start by analyzing the modality alignment behaviors of
task-specific expert models for understanding and generation, as well as
current unified models. Our analysis reveals a crucial observation:
understanding tasks benefit from a progressively increasing modality alignment
across network depth, which helps build up semantic information for better
comprehension; In contrast, generation tasks follow a different trend: modality
alignment increases in the early layers but decreases in the deep layers to
recover spatial details. These divergent alignment patterns create a
fundamental conflict in fully shared Transformer backbones, where a uniform
representational flow often leads to performance compromises across two tasks.
Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture
that shares the shallow layers for cross-task representation learning, while
employing task-specific branches in deeper layers to avoid task interference.
This design effectively balances shared learning and task specialization.
Through extensive ablation experiments, we demonstrate that Unifork
consistently outperforms conventional fully shared Transformer architectures,
and achieves performance on par with or better than task-specific models.
68591c310e4ad7e219758306
Teng Li
claimed_verified
2025-06-24T08:09:36.113Z
677a3eb68b252e571a1fce1a
/avatars/f2260ef6ade5cb22606516f26e872b57.svg
Teng LI
user
tliby
68591c310e4ad7e219758307
Quanfeng Lu
68591c310e4ad7e219758308
Lirui Zhao
68591c310e4ad7e219758309
Hao Li
68591c310e4ad7e21975830a
Xizhou Zhu
68591c310e4ad7e21975830b
Yu Qiao
68591c310e4ad7e21975830c
Jun Zhang
68591c310e4ad7e21975830d
Wenqi Shao
UniFork: Exploring Modality Alignment for Unified Multimodal
  Understanding and Generation
Unified image understanding and generation has emerged as a promising
paradigm in multimodal artificial intelligence. Despite recent progress, the
optimal architectural design for such unified models remains an open challenge.
In this work, we start by analyzing the modality alignment behaviors of
task-specific expert models for understanding and generation, as well as
current unified models. Our analysis reveals a crucial observation:
understanding tasks benefit from a progressively increasing modality alignment
across network depth, which helps build up semantic information for better
comprehension; In contrast, generation tasks follow a different trend: modality
alignment increases in the early layers but decreases in the deep layers to
recover spatial details. These divergent alignment patterns create a
fundamental conflict in fully shared Transformer backbones, where a uniform
representational flow often leads to performance compromises across two tasks.
Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture
that shares the shallow layers for cross-task representation learning, while
employing task-specific branches in deeper layers to avoid task interference.
This design effectively balances shared learning and task specialization.
Through extensive ablation experiments, we demonstrate that Unifork
consistently outperforms conventional fully shared Transformer architectures,
and achieves performance on par with or better than task-specific models.
A Y-shaped architecture, UniFork, balances shared learning and task specialization for unified image understanding and generation, outperforming conventional fully shared Transformer models.
modality alignment network depth semantic information spatial details Transformer backbones Y-shaped architecture task-specific branches task interference ablation experiments fully shared Transformer architectures task-specific models