title: OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
summary: We introduce OmniFlow, a novel generative model designed for any-to-any
  generation tasks such as text-to-image, text-to-audio, and audio-to-image
  synthesis. OmniFlow advances the rectified flow (RF) framework used in
  text-to-image models to handle the joint distribution of multiple modalities.
  It outperforms previous any-to-any models on a wide range of tasks, such as
  text-to-image and text-to-audio synthesis. Our work offers three key
  contributions: First, we extend RF to a multi-modal setting and introduce a
  novel guidance mechanism, enabling users to flexibly control the alignment
  between different modalities in the generated outputs. Second, we propose a
  novel architecture that extends the text-to-image MMDiT architecture of Stable
  Diffusion 3 and enables audio and text generation. The extended modules can be
  efficiently pretrained individually and merged with the vanilla text-to-image
  MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
  choices of rectified flow transformers for large-scale audio and text
  generation, providing valuable insights into optimizing performance across
  diverse modalities. The Code will be available at
  https://github.com/jacklishufan/OmniFlows.
paper_authors._id: 67507dcb10a0006992dd4612
paper_authors.name: Shufan Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:57:08.061Z
paper_authors._id: 6310531914aa81e1044363ed
paper_authors.avatarUrl: /avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
paper_authors.fullname: Shufan Li
paper_authors.type: user
paper_authors.user: jacklishufan
paper_authors._id: 67507dcb10a0006992dd4613
paper_authors.name: Konstantinos Kallidromitis
paper_authors._id: 67507dcb10a0006992dd4614
paper_authors.name: Akash Gokul
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:57:17.786Z
paper_authors._id: 63093856b105f8675bd1c783
paper_authors.avatarUrl: /avatars/505a7b99893330a36a136d13fa99e05e.svg
paper_authors.fullname: Akash Gokul
paper_authors.type: user
paper_authors.user: akashgokul
paper_authors._id: 67507dcb10a0006992dd4615
paper_authors.name: Zichun Liao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:57:23.985Z
paper_authors._id: 6530c763471261a412485382
paper_authors.avatarUrl: /avatars/75b52782fd41beae608c1e0232b5a5b0.svg
paper_authors.fullname: Zichun Liao
paper_authors.type: user
paper_authors.user: shinosaikou
paper_authors._id: 67507dcb10a0006992dd4616
paper_authors.name: Yusuke Kato
paper_authors._id: 67507dcb10a0006992dd4617
paper_authors.name: Kazuki Kozuka
paper_authors._id: 67507dcb10a0006992dd4618
paper_authors.name: Aditya Grover
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:57:37.291Z
paper_authors._id: 65d38c1e5d5749637491679b
paper_authors.avatarUrl: /avatars/010a6588f382d6d4394216549a2385e3.svg
paper_authors.fullname: Aditya Grover
paper_authors.type: user
paper_authors.user: adityagrover
paper_title: OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
paper_summary: We introduce OmniFlow, a novel generative model designed for any-to-any
  generation tasks such as text-to-image, text-to-audio, and audio-to-image
  synthesis. OmniFlow advances the rectified flow (RF) framework used in
  text-to-image models to handle the joint distribution of multiple modalities.
  It outperforms previous any-to-any models on a wide range of tasks, such as
  text-to-image and text-to-audio synthesis. Our work offers three key
  contributions: First, we extend RF to a multi-modal setting and introduce a
  novel guidance mechanism, enabling users to flexibly control the alignment
  between different modalities in the generated outputs. Second, we propose a
  novel architecture that extends the text-to-image MMDiT architecture of Stable
  Diffusion 3 and enables audio and text generation. The extended modules can be
  efficiently pretrained individually and merged with the vanilla text-to-image
  MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
  choices of rectified flow transformers for large-scale audio and text
  generation, providing valuable insights into optimizing performance across
  diverse modalities. The Code will be available at
  https://github.com/jacklishufan/OmniFlows.
paper_ai_summary: OmniFlow extends rectified flow to a multi-modal generative model, enhancing text-to-image, text-to-audio, and audio-to-image synthesis with novel guidance and architectural improvements.
paper_ai_keywords: OmniFlow rectified flow (RF) multi-modal generative model guidance mechanism text-to-image text-to-audio audio-to-image MMDiT Stable Diffusion 3 fine-tuning rectified flow transformers