title: CoDeF: Content Deformation Fields for Temporally Consistent Video
    Processing
summary: We present the content deformation field CoDeF as a new type of video
  representation, which consists of a canonical content field aggregating the
  static contents in the entire video and a temporal deformation field recording
  the transformations from the canonical image (i.e., rendered from the canonical
  content field) to each individual frame along the time axis.Given a target
  video, these two fields are jointly optimized to reconstruct it through a
  carefully tailored rendering pipeline.We advisedly introduce some
  regularizations into the optimization process, urging the canonical content
  field to inherit semantics (e.g., the object shape) from the video.With such a
  design, CoDeF naturally supports lifting image algorithms for video processing,
  in the sense that one can apply an image algorithm to the canonical image and
  effortlessly propagate the outcomes to the entire video with the aid of the
  temporal deformation field.We experimentally show that CoDeF is able to lift
  image-to-image translation to video-to-video translation and lift keypoint
  detection to keypoint tracking without any training.More importantly, thanks to
  our lifting strategy that deploys the algorithms on only one image, we achieve
  superior cross-frame consistency in processed videos compared to existing
  video-to-video translation approaches, and even manage to track non-rigid
  objects like water and smog.Project page can be found at
  https://qiuyu96.github.io/CoDeF/.
paper_authors._id: 64dd45b3c9a04fc4a5f6638a
paper_authors.name: Hao Ouyang
paper_authors._id: 64dd45b3c9a04fc4a5f6638b
paper_authors.name: Qiuyu Wang
paper_authors._id: 64dd45b3c9a04fc4a5f6638c
paper_authors.name: Yuxi Xiao
paper_authors._id: 64dd45b3c9a04fc4a5f6638d
paper_authors.name: Qingyan Bai
paper_authors._id: 64dd45b3c9a04fc4a5f6638e
paper_authors.name: Juntao Zhang
paper_authors._id: 64dd45b3c9a04fc4a5f6638f
paper_authors.name: Kecheng Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-20T14:13:57.205Z
paper_authors._id: 64252045a4f3051f54dd1d53
paper_authors.avatarUrl: /avatars/0e423a3291091be3b4736a14da3ce495.svg
paper_authors.fullname: kecheng zheng
paper_authors.type: user
paper_authors.user: zkcys001
paper_authors._id: 64dd45b3c9a04fc4a5f66390
paper_authors.name: Xiaowei Zhou
paper_authors._id: 64dd45b3c9a04fc4a5f66391
paper_authors.name: Qifeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-20T14:14:48.369Z
paper_authors._id: 6467b121e7a6a374fd19b44b
paper_authors.avatarUrl: /avatars/3f2874d58986d651aef55e3408b05700.svg
paper_authors.fullname: Qifeng Chen
paper_authors.type: user
paper_authors.user: cqf
paper_authors._id: 64dd45b3c9a04fc4a5f66392
paper_authors.name: Yujun Shen
paper_title: CoDeF: Content Deformation Fields for Temporally Consistent Video
    Processing
paper_summary: We present the content deformation field CoDeF as a new type of video
  representation, which consists of a canonical content field aggregating the
  static contents in the entire video and a temporal deformation field recording
  the transformations from the canonical image (i.e., rendered from the canonical
  content field) to each individual frame along the time axis.Given a target
  video, these two fields are jointly optimized to reconstruct it through a
  carefully tailored rendering pipeline.We advisedly introduce some
  regularizations into the optimization process, urging the canonical content
  field to inherit semantics (e.g., the object shape) from the video.With such a
  design, CoDeF naturally supports lifting image algorithms for video processing,
  in the sense that one can apply an image algorithm to the canonical image and
  effortlessly propagate the outcomes to the entire video with the aid of the
  temporal deformation field.We experimentally show that CoDeF is able to lift
  image-to-image translation to video-to-video translation and lift keypoint
  detection to keypoint tracking without any training.More importantly, thanks to
  our lifting strategy that deploys the algorithms on only one image, we achieve
  superior cross-frame consistency in processed videos compared to existing
  video-to-video translation approaches, and even manage to track non-rigid
  objects like water and smog.Project page can be found at
  https://qiuyu96.github.io/CoDeF/.
paper_ai_keywords: canonical content field content deformation field (CoDeF) temporal deformation field canonical image rendering pipeline regularizations semantics object shape image algorithms video processing image-to-image translation video-to-video translation keypoint detection keypoint tracking cross-frame consistency non-rigid objects