Jump Cut Smoothing for Talking Heads
A jump cut offers an abrupt, sometimes unwanted change in the viewing
experience. We present a novel framework for smoothing these jump cuts, in the
context of talking head videos. We leverage the appearance of the subject from
the other source frames in the video, fusing it with a mid-level representation
driven by DensePose keypoints and face landmarks. To achieve motion, we
interpolate the keypoints and landmarks between the end frames around the cut.
We then use an image translation network from the keypoints and source frames,
to synthesize pixels. Because keypoints can contain errors, we propose a
cross-modal attention scheme to select and pick the most appropriate source
amongst multiple options for each key point. By leveraging this mid-level
representation, our method can achieve stronger results than a strong video
interpolation baseline. We demonstrate our method on various jump cuts in the
talking head videos, such as cutting filler words, pauses, and even random
cuts. Our experiments show that we can achieve seamless transitions, even in
the challenging cases where the talking head rotates or moves drastically in
the jump cut.
659e2024962de309f03e330a
Xiaojuan Wang
admin_assigned
2024-01-10T08:53:46.072Z
643ee9a6022a0350efd745d1
/avatars/c8c17689ed88b1966d2a36361e3f83cc.svg
Xiaojuan Wang
user
xiaojwan
659e2024962de309f03e330b
Taesung Park
admin_assigned
2024-01-10T08:54:14.489Z
637563c59ef4e3b87a60d503
/avatars/8264de115e6305e4720c78fc7dce7f8d.svg
Taesung Park
user
taesungp
659e2024962de309f03e330c
Yang Zhou
659e2024962de309f03e330d
Eli Shechtman
659e2024962de309f03e330e
Richard Zhang
Jump Cut Smoothing for Talking Heads
A jump cut offers an abrupt, sometimes unwanted change in the viewing
experience. We present a novel framework for smoothing these jump cuts, in the
context of talking head videos. We leverage the appearance of the subject from
the other source frames in the video, fusing it with a mid-level representation
driven by DensePose keypoints and face landmarks. To achieve motion, we
interpolate the keypoints and landmarks between the end frames around the cut.
We then use an image translation network from the keypoints and source frames,
to synthesize pixels. Because keypoints can contain errors, we propose a
cross-modal attention scheme to select and pick the most appropriate source
amongst multiple options for each key point. By leveraging this mid-level
representation, our method can achieve stronger results than a strong video
interpolation baseline. We demonstrate our method on various jump cuts in the
talking head videos, such as cutting filler words, pauses, and even random
cuts. Our experiments show that we can achieve seamless transitions, even in
the challenging cases where the talking head rotates or moves drastically in
the jump cut.
A framework for smoothing jump cuts in talking head videos using DensePose keypoints, face landmarks, and cross-modal attention for seamless transitions.
DensePose keypoints face landmarks image translation network cross-modal attention video interpolation talking head videos jump cuts seamless transitions