(cid:15)

5e − 5
1024

1000
Linear
0.00085
0.0120

DDIM

50
1.0
2.0

A.2 Video LDM Architecture

Except for the base image UNet architecture, we also add temporal attention and temporal shift [2]
before each residual block. Following VDM [21], the temporal attention is a transformer attention
module where we ﬂatten the height and width dimension to batch size dimension and the self-attention
is performed on the time dimension. The temporal shift is illustrated in Fig. 6 where we ﬁrst split
channels into k chunks. Then, we shift the channel dimension numbered 0 to k − 1 by temporal
dimension from 0 to k − 1 times respectively. Eventually, we concatenate the shifted chunks by
the hidden dimension. Note that we use k = 3 in the illustration for simplicity but k = 8 in our
implementation. We then add a convolution layer before the temporal shift module. Finally, we use
residual connection [18] and add the output to the input before the convolution layer.

B Model Training
Any-to-Any Generation via Composable Diffusion