OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
We introduce OmniFlow, a novel generative model designed for any-to-any
generation tasks such as text-to-image, text-to-audio, and audio-to-image
synthesis. OmniFlow advances the rectified flow (RF) framework used in
text-to-image models to handle the joint distribution of multiple modalities.
It outperforms previous any-to-any models on a wide range of tasks, such as
text-to-image and text-to-audio synthesis. Our work offers three key
contributions: First, we extend RF to a multi-modal setting and introduce a
novel guidance mechanism, enabling users to flexibly control the alignment
between different modalities in the generated outputs. Second, we propose a
novel architecture that extends the text-to-image MMDiT architecture of Stable
Diffusion 3 and enables audio and text generation. The extended modules can be
efficiently pretrained individually and merged with the vanilla text-to-image
MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
choices of rectified flow transformers for large-scale audio and text
generation, providing valuable insights into optimizing performance across
diverse modalities. The Code will be available at
https://github.com/jacklishufan/OmniFlows.
67507dcb10a0006992dd4612
Shufan Li
admin_assigned
2024-12-16T13:57:08.061Z
6310531914aa81e1044363ed
/avatars/ae7767e591cb7199ea2f62d2db89fc7f.svg
Shufan Li
user
jacklishufan
67507dcb10a0006992dd4613
Konstantinos Kallidromitis
67507dcb10a0006992dd4614
Akash Gokul
admin_assigned
2024-12-16T13:57:17.786Z
63093856b105f8675bd1c783
/avatars/505a7b99893330a36a136d13fa99e05e.svg
Akash Gokul
user
akashgokul
67507dcb10a0006992dd4615
Zichun Liao
admin_assigned
2024-12-16T13:57:23.985Z
6530c763471261a412485382
/avatars/75b52782fd41beae608c1e0232b5a5b0.svg
Zichun Liao
user
shinosaikou
67507dcb10a0006992dd4616
Yusuke Kato
67507dcb10a0006992dd4617
Kazuki Kozuka
67507dcb10a0006992dd4618
Aditya Grover
admin_assigned
2024-12-16T13:57:37.291Z
65d38c1e5d5749637491679b
/avatars/010a6588f382d6d4394216549a2385e3.svg
Aditya Grover
user
adityagrover
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
We introduce OmniFlow, a novel generative model designed for any-to-any
generation tasks such as text-to-image, text-to-audio, and audio-to-image
synthesis. OmniFlow advances the rectified flow (RF) framework used in
text-to-image models to handle the joint distribution of multiple modalities.
It outperforms previous any-to-any models on a wide range of tasks, such as
text-to-image and text-to-audio synthesis. Our work offers three key
contributions: First, we extend RF to a multi-modal setting and introduce a
novel guidance mechanism, enabling users to flexibly control the alignment
between different modalities in the generated outputs. Second, we propose a
novel architecture that extends the text-to-image MMDiT architecture of Stable
Diffusion 3 and enables audio and text generation. The extended modules can be
efficiently pretrained individually and merged with the vanilla text-to-image
MMDiT for fine-tuning. Lastly, we conduct a comprehensive study on the design
choices of rectified flow transformers for large-scale audio and text
generation, providing valuable insights into optimizing performance across
diverse modalities. The Code will be available at
https://github.com/jacklishufan/OmniFlows.
OmniFlow extends rectified flow to a multi-modal generative model, enhancing text-to-image, text-to-audio, and audio-to-image synthesis with novel guidance and architectural improvements.
OmniFlow rectified flow (RF) multi-modal generative model guidance mechanism text-to-image text-to-audio audio-to-image MMDiT Stable Diffusion 3 fine-tuning rectified flow transformers