RepVideo: Rethinking Cross-Layer Representation for Video Generation
Video generation has achieved remarkable progress with the introduction of
diffusion models, which have significantly improved the quality of generated
videos. However, recent research has primarily focused on scaling up model
training, while offering limited insights into the direct impact of
representations on the video generation process. In this paper, we initially
investigate the characteristics of features in intermediate layers, finding
substantial variations in attention maps across different layers. These
variations lead to unstable semantic representations and contribute to
cumulative differences between features, which ultimately reduce the similarity
between adjacent frames and negatively affect temporal coherence. To address
this, we propose RepVideo, an enhanced representation framework for
text-to-video diffusion models. By accumulating features from neighboring
layers to form enriched representations, this approach captures more stable
semantic information. These enhanced representations are then used as inputs to
the attention mechanism, thereby improving semantic expressiveness while
ensuring feature consistency across adjacent frames. Extensive experiments
demonstrate that our RepVideo not only significantly enhances the ability to
generate accurate spatial appearances, such as capturing complex spatial
relationships between multiple objects, but also improves temporal consistency
in video generation.
6788945e2b5050a9154d939d
Chenyang Si
admin_assigned
2025-01-16T08:49:14.615Z
635f8ed47c05eb9f59963d3a
https://cdn-avatars.huggingface.co/v1/production/uploads/635f8ed47c05eb9f59963d3a/uQf4p9N9pSaFy87Wg9v4k.jpeg
ChenyangSi
user
ChenyangSi
6788945e2b5050a9154d939e
Weichen Fan
admin_assigned
2025-01-16T08:49:38.866Z
6481764e8af4675862efb22e
/avatars/fc2e076bc861693f598a528a068a696e.svg
weichenfan
user
weepiess2383
6788945e2b5050a9154d939f
Zhengyao Lv
admin_assigned
2025-01-16T08:49:45.309Z
645aff5121ab438e732c47c1
/avatars/23b2a853139b0f2ae1fa88e2bd4e0056.svg
Zhengyao Lv
user
cszy98
6788945e2b5050a9154d93a0
Ziqi Huang
admin_assigned
2025-01-16T08:49:51.123Z
60efe7fa0d920bc7805cada5
https://cdn-avatars.huggingface.co/v1/production/uploads/60efe7fa0d920bc7805cada5/2LBrJBjSCOP5ilZIpWLHl.png
Ziqi Huang
user
Ziqi
6788945e2b5050a9154d93a1
Yu Qiao
6788945e2b5050a9154d93a2
Ziwei Liu
admin_assigned
2025-01-16T08:49:58.339Z
62ab1ac1d48b4d8b048a3473
https://cdn-avatars.huggingface.co/v1/production/uploads/1656826685333-62ab1ac1d48b4d8b048a3473.png
Ziwei Liu
user
liuziwei7
RepVideo: Rethinking Cross-Layer Representation for Video Generation
Video generation has achieved remarkable progress with the introduction of
diffusion models, which have significantly improved the quality of generated
videos. However, recent research has primarily focused on scaling up model
training, while offering limited insights into the direct impact of
representations on the video generation process. In this paper, we initially
investigate the characteristics of features in intermediate layers, finding
substantial variations in attention maps across different layers. These
variations lead to unstable semantic representations and contribute to
cumulative differences between features, which ultimately reduce the similarity
between adjacent frames and negatively affect temporal coherence. To address
this, we propose RepVideo, an enhanced representation framework for
text-to-video diffusion models. By accumulating features from neighboring
layers to form enriched representations, this approach captures more stable
semantic information. These enhanced representations are then used as inputs to
the attention mechanism, thereby improving semantic expressiveness while
ensuring feature consistency across adjacent frames. Extensive experiments
demonstrate that our RepVideo not only significantly enhances the ability to
generate accurate spatial appearances, such as capturing complex spatial
relationships between multiple objects, but also improves temporal consistency
in video generation.
RepVideo enhances video generation by stabilizing semantic representations through enriched feature accumulation, improving spatial accuracy and temporal coherence in text-to-video diffusion models.
diffusion models attention maps semantic representations temporal coherence RepVideo feature accumulation spatial appearances temporal consistency