title: Jump Cut Smoothing for Talking Heads
summary: A jump cut offers an abrupt, sometimes unwanted change in the viewing
  experience. We present a novel framework for smoothing these jump cuts, in the
  context of talking head videos. We leverage the appearance of the subject from
  the other source frames in the video, fusing it with a mid-level representation
  driven by DensePose keypoints and face landmarks. To achieve motion, we
  interpolate the keypoints and landmarks between the end frames around the cut.
  We then use an image translation network from the keypoints and source frames,
  to synthesize pixels. Because keypoints can contain errors, we propose a
  cross-modal attention scheme to select and pick the most appropriate source
  amongst multiple options for each key point. By leveraging this mid-level
  representation, our method can achieve stronger results than a strong video
  interpolation baseline. We demonstrate our method on various jump cuts in the
  talking head videos, such as cutting filler words, pauses, and even random
  cuts. Our experiments show that we can achieve seamless transitions, even in
  the challenging cases where the talking head rotates or moves drastically in
  the jump cut.
paper_authors._id: 659e2024962de309f03e330a
paper_authors.name: Xiaojuan Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T08:53:46.072Z
paper_authors._id: 643ee9a6022a0350efd745d1
paper_authors.avatarUrl: /avatars/c8c17689ed88b1966d2a36361e3f83cc.svg
paper_authors.fullname: Xiaojuan Wang
paper_authors.type: user
paper_authors.user: xiaojwan
paper_authors._id: 659e2024962de309f03e330b
paper_authors.name: Taesung Park
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-10T08:54:14.489Z
paper_authors._id: 637563c59ef4e3b87a60d503
paper_authors.avatarUrl: /avatars/8264de115e6305e4720c78fc7dce7f8d.svg
paper_authors.fullname: Taesung Park
paper_authors.type: user
paper_authors.user: taesungp
paper_authors._id: 659e2024962de309f03e330c
paper_authors.name: Yang Zhou
paper_authors._id: 659e2024962de309f03e330d
paper_authors.name: Eli Shechtman
paper_authors._id: 659e2024962de309f03e330e
paper_authors.name: Richard Zhang
paper_title: Jump Cut Smoothing for Talking Heads
paper_summary: A jump cut offers an abrupt, sometimes unwanted change in the viewing
  experience. We present a novel framework for smoothing these jump cuts, in the
  context of talking head videos. We leverage the appearance of the subject from
  the other source frames in the video, fusing it with a mid-level representation
  driven by DensePose keypoints and face landmarks. To achieve motion, we
  interpolate the keypoints and landmarks between the end frames around the cut.
  We then use an image translation network from the keypoints and source frames,
  to synthesize pixels. Because keypoints can contain errors, we propose a
  cross-modal attention scheme to select and pick the most appropriate source
  amongst multiple options for each key point. By leveraging this mid-level
  representation, our method can achieve stronger results than a strong video
  interpolation baseline. We demonstrate our method on various jump cuts in the
  talking head videos, such as cutting filler words, pauses, and even random
  cuts. Our experiments show that we can achieve seamless transitions, even in
  the challenging cases where the talking head rotates or moves drastically in
  the jump cut.
paper_ai_summary: A framework for smoothing jump cuts in talking head videos using DensePose keypoints, face landmarks, and cross-modal attention for seamless transitions.
paper_ai_keywords: DensePose keypoints face landmarks image translation network cross-modal attention video interpolation talking head videos jump cuts seamless transitions