TextEncoderVisionEncoderAudioEncoder(2) Diffusion Step(s)Text UNetVision UNetAudio UNet(1) ConditioningL2LossL2LossVisionEnvironmentEncoderAudioEnvironmentEncoderVision UNetAudio UNetTextEnvironmentEncoderText UNetBridgingAlignmentVision UNetAudio UNetDecodeDecodeInferenceStage 2JointGenerationVisionEncoderTextEncodern diffusion stepsA panda enjoyingher breakfast atthe table (b)(c)(3) Latent Alignment at Step tStage 1ComposableConditioning(a)TextEncoderTextEncoderVisionEncoderAudioEncoderText UNetVision UNetAudio UNetL2Loss(1) Conditioning Alignment(2) Single Diffuser TrainingL2LossBridging AlignmentVideoEncoderAudioEncoderImageEncoderIn LDM, an autoencoder is ﬁrst trained to reconstruct x, i.e., ˆx = D(E(x)), where E and D denote
the encoder and decoder, respectively. The latent variable z = E(x) is iteratively diffused over
time steps t based on a variance schedule β1, . . . , βT , i.e., q(zt|zt−1) = N (zt;
1 − βtzt−1, βtI)
[20, 45].
Any-to-Any Generation via Composable Diffusion