title: DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in
    Multimodal Cycles
summary: Adapting generative models to specific domains presents an effective solution
  for satisfying specialized requirements. However, adapting to some complex
  domains remains challenging, especially when these domains require substantial
  paired data to capture the targeted distributions. Since unpaired data from a
  single modality, such as vision or language, is more readily available, we
  utilize the bidirectional mappings between vision and language learned by the
  unified generative model to enable training on unpaired data for domain
  adaptation. Specifically, we propose DoraCycle, which integrates two multimodal
  cycles: text-to-image-to-text and image-to-text-to-image. The model is
  optimized through cross-entropy loss computed at the cycle endpoints, where
  both endpoints share the same modality. This facilitates self-evolution of the
  model without reliance on annotated text-image pairs. Experimental results
  demonstrate that for tasks independent of paired knowledge, such as
  stylization, DoraCycle can effectively adapt the unified model using only
  unpaired data. For tasks involving new paired knowledge, such as specific
  identities, a combination of a small set of paired image-text examples and
  larger-scale unpaired data is sufficient for effective domain-oriented
  adaptation. The code will be released at https://github.com/showlab/DoraCycle.
paper_authors._id: 67c902cd4ab18796fcd2cbda
paper_authors.name: Rui Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-06T09:26:52.526Z
paper_authors._id: 652b83b73b5997ed71a310f2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652b83b73b5997ed71a310f2/ipCpdeHUp4-0OmRz5z8IW.png
paper_authors.fullname: Rui Zhao
paper_authors.type: user
paper_authors.user: ruizhaocv
paper_authors._id: 67c902cd4ab18796fcd2cbdb
paper_authors.name: Weijia Mao
paper_authors._id: 67c902cd4ab18796fcd2cbdc
paper_authors.name: Mike Zheng Shou
paper_title: DoraCycle: Domain-Oriented Adaptation of Unified Generative Model in
    Multimodal Cycles
paper_summary: Adapting generative models to specific domains presents an effective solution
  for satisfying specialized requirements. However, adapting to some complex
  domains remains challenging, especially when these domains require substantial
  paired data to capture the targeted distributions. Since unpaired data from a
  single modality, such as vision or language, is more readily available, we
  utilize the bidirectional mappings between vision and language learned by the
  unified generative model to enable training on unpaired data for domain
  adaptation. Specifically, we propose DoraCycle, which integrates two multimodal
  cycles: text-to-image-to-text and image-to-text-to-image. The model is
  optimized through cross-entropy loss computed at the cycle endpoints, where
  both endpoints share the same modality. This facilitates self-evolution of the
  model without reliance on annotated text-image pairs. Experimental results
  demonstrate that for tasks independent of paired knowledge, such as
  stylization, DoraCycle can effectively adapt the unified model using only
  unpaired data. For tasks involving new paired knowledge, such as specific
  identities, a combination of a small set of paired image-text examples and
  larger-scale unpaired data is sufficient for effective domain-oriented
  adaptation. The code will be released at https://github.com/showlab/DoraCycle.
paper_ai_summary: DoraCycle adapts generative models to specific domains using unpaired data through text-to-image and image-to-text cycles, enabling both independent and paired knowledge tasks.
paper_ai_keywords: generative models domain adaptation unpaired data bidirectional mappings vision language unified generative model multimodal cycles cross-entropy loss stylization paired image-text examples domain-oriented adaptation