title: Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
summary: Diffusion models, and their generalization, flow matching, have had a
  remarkable impact on the field of media generation. Here, the conventional
  approach is to learn the complex mapping from a simple source distribution of
  Gaussian noise to the target media distribution. For cross-modal tasks such as
  text-to-image generation, this same mapping from noise to image is learnt
  whilst including a conditioning mechanism in the model. One key and thus far
  relatively unexplored feature of flow matching is that, unlike Diffusion
  models, they are not constrained for the source distribution to be noise.
  Hence, in this paper, we propose a paradigm shift, and ask the question of
  whether we can instead train flow matching models to learn a direct mapping
  from the distribution of one modality to the distribution of another, thus
  obviating the need for both the noise distribution and conditioning mechanism.
  We present a general and simple framework, CrossFlow, for cross-modal flow
  matching. We show the importance of applying Variational Encoders to the input
  data, and introduce a method to enable Classifier-free guidance. Surprisingly,
  for text-to-image, CrossFlow with a vanilla transformer without cross attention
  slightly outperforms standard flow matching, and we show that it scales better
  with training steps and model size, while also allowing for interesting latent
  arithmetic which results in semantically meaningful edits in the output space.
  To demonstrate the generalizability of our approach, we also show that
  CrossFlow is on par with or outperforms the state-of-the-art for various
  cross-modal / intra-modal mapping tasks, viz. image captioning, depth
  estimation, and image super-resolution. We hope this paper contributes to
  accelerating progress in cross-modal media generation.
paper_authors._id: 6764dd4d469e108330dc958a
paper_authors.name: Qihao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:21:57.934Z
paper_authors._id: 639f1e519f1f2baab2f00d22
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639f1e519f1f2baab2f00d22/pFjd51WZuVZ3A11rItvmk.jpeg
paper_authors.fullname: Qihao Liu
paper_authors.type: user
paper_authors.user: QHL067
paper_authors._id: 6764dd4d469e108330dc958b
paper_authors.name: Xi Yin
paper_authors._id: 6764dd4d469e108330dc958c
paper_authors.name: Alan Yuille
paper_authors._id: 6764dd4d469e108330dc958d
paper_authors.name: Andrew Brown
paper_authors._id: 6764dd4d469e108330dc958e
paper_authors.name: Mannat Singh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-20T09:22:34.696Z
paper_authors._id: 650fece35877b1c077298b81
paper_authors.avatarUrl: /avatars/be9c948ecf9c413ad4c8aa6598d3ba86.svg
paper_authors.fullname: Mannat Singh
paper_authors.type: user
paper_authors.user: mannatsingh
paper_title: Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
paper_summary: Diffusion models, and their generalization, flow matching, have had a
  remarkable impact on the field of media generation. Here, the conventional
  approach is to learn the complex mapping from a simple source distribution of
  Gaussian noise to the target media distribution. For cross-modal tasks such as
  text-to-image generation, this same mapping from noise to image is learnt
  whilst including a conditioning mechanism in the model. One key and thus far
  relatively unexplored feature of flow matching is that, unlike Diffusion
  models, they are not constrained for the source distribution to be noise.
  Hence, in this paper, we propose a paradigm shift, and ask the question of
  whether we can instead train flow matching models to learn a direct mapping
  from the distribution of one modality to the distribution of another, thus
  obviating the need for both the noise distribution and conditioning mechanism.
  We present a general and simple framework, CrossFlow, for cross-modal flow
  matching. We show the importance of applying Variational Encoders to the input
  data, and introduce a method to enable Classifier-free guidance. Surprisingly,
  for text-to-image, CrossFlow with a vanilla transformer without cross attention
  slightly outperforms standard flow matching, and we show that it scales better
  with training steps and model size, while also allowing for interesting latent
  arithmetic which results in semantically meaningful edits in the output space.
  To demonstrate the generalizability of our approach, we also show that
  CrossFlow is on par with or outperforms the state-of-the-art for various
  cross-modal / intra-modal mapping tasks, viz. image captioning, depth
  estimation, and image super-resolution. We hope this paper contributes to
  accelerating progress in cross-modal media generation.
paper_ai_summary: CrossFlow proposes a direct mapping between modalities for cross-modal media generation using flow matching, outperforming standard methods in various tasks.
paper_ai_keywords: diffusion models flow matching Gaussian noise cross-modal tasks text-to-image generation noise distribution conditioning mechanism CrossFlow Variational Encoders Classifier-free guidance image captioning depth estimation image super-resolution