DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in
  Multimodal Cycles
Adapting generative models to specific domains presents an effective solution
for satisfying specialized requirements. However, adapting to some complex
domains remains challenging, especially when these domains require substantial
paired data to capture the targeted distributions. Since unpaired data from a
single modality, such as vision or language, is more readily available, we
utilize the bidirectional mappings between vision and language learned by the
unified generative model to enable training on unpaired data for domain
adaptation. Specifically, we propose DoraCycle, which integrates two multimodal
cycles: text-to-image-to-text and image-to-text-to-image. The model is
optimized through cross-entropy loss computed at the cycle endpoints, where
both endpoints share the same modality. This facilitates self-evolution of the
model without reliance on annotated text-image pairs. Experimental results
demonstrate that for tasks independent of paired knowledge, such as
stylization, DoraCycle can effectively adapt the unified model using only
unpaired data. For tasks involving new paired knowledge, such as specific
identities, a combination of a small set of paired image-text examples and
larger-scale unpaired data is sufficient for effective domain-oriented
adaptation. The code will be released at https://github.com/showlab/DoraCycle.
67c902cd4ab18796fcd2cbda
Rui Zhao
claimed_verified
2025-03-06T09:26:52.526Z
652b83b73b5997ed71a310f2
https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
Rui Zhao
user
ruizhaocv
67c902cd4ab18796fcd2cbdb
Weijia Mao
67c902cd4ab18796fcd2cbdc
Mike Zheng Shou
DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in
  Multimodal Cycles
Adapting generative models to specific domains presents an effective solution
for satisfying specialized requirements. However, adapting to some complex
domains remains challenging, especially when these domains require substantial
paired data to capture the targeted distributions. Since unpaired data from a
single modality, such as vision or language, is more readily available, we
utilize the bidirectional mappings between vision and language learned by the
unified generative model to enable training on unpaired data for domain
adaptation. Specifically, we propose DoraCycle, which integrates two multimodal
cycles: text-to-image-to-text and image-to-text-to-image. The model is
optimized through cross-entropy loss computed at the cycle endpoints, where
both endpoints share the same modality. This facilitates self-evolution of the
model without reliance on annotated text-image pairs. Experimental results
demonstrate that for tasks independent of paired knowledge, such as
stylization, DoraCycle can effectively adapt the unified model using only
unpaired data. For tasks involving new paired knowledge, such as specific
identities, a combination of a small set of paired image-text examples and
larger-scale unpaired data is sufficient for effective domain-oriented
adaptation. The code will be released at https://github.com/showlab/DoraCycle.
DoraCycle adapts generative models to specific domains using unpaired data through text-to-image and image-to-text cycles, enabling both independent and paired knowledge tasks.
generative models domain adaptation unpaired data bidirectional mappings vision language unified generative model multimodal cycles cross-entropy loss stylization paired image-text examples domain-oriented adaptation