To enable our model to condition on any combination of input/prompt modalities, we align the prompt
encoder of text, image, video and audio (denoted by Ct, Ci, Cv, and Ca, respectively) to project the
input from any modality into the same space. Multimodal conditioning can then be conveniently
m αmC(m)
m αm = 1. Through simple weighted interpolation of aligned
embeddings, we enable models trained with single-conditioning (i.e., with only one input) to perform
zero-shot multi-conditioning (i.e., with multiple inputs). This process is illustrated in Fig. 2 (a)(2).
Optimizing all four prompt encoders simultaneously in a combinatorial manner is computationally
heavy, with O(n2) pairs. Additionally, for certain dual modalities, well-aligned paired datasets are
limited or unavailable e.g., image-audio pairs. To address this challenge, we propose a simple and
effective technique called "Bridging Alignment" to efﬁciently align conditional encoders. As shown
Any-to-Any Generation via Composable Diffusion