CoDeF: Content Deformation Fields for Temporally Consistent Video
  Processing
We present the content deformation field CoDeF as a new type of video
representation, which consists of a canonical content field aggregating the
static contents in the entire video and a temporal deformation field recording
the transformations from the canonical image (i.e., rendered from the canonical
content field) to each individual frame along the time axis.Given a target
video, these two fields are jointly optimized to reconstruct it through a
carefully tailored rendering pipeline.We advisedly introduce some
regularizations into the optimization process, urging the canonical content
field to inherit semantics (e.g., the object shape) from the video.With such a
design, CoDeF naturally supports lifting image algorithms for video processing,
in the sense that one can apply an image algorithm to the canonical image and
effortlessly propagate the outcomes to the entire video with the aid of the
temporal deformation field.We experimentally show that CoDeF is able to lift
image-to-image translation to video-to-video translation and lift keypoint
detection to keypoint tracking without any training.More importantly, thanks to
our lifting strategy that deploys the algorithms on only one image, we achieve
superior cross-frame consistency in processed videos compared to existing
video-to-video translation approaches, and even manage to track non-rigid
objects like water and smog.Project page can be found at
https://qiuyu96.github.io/CoDeF/.
64dd45b3c9a04fc4a5f6638a
Hao Ouyang
64dd45b3c9a04fc4a5f6638b
Qiuyu Wang
64dd45b3c9a04fc4a5f6638c
Yuxi Xiao
claimed_verified
2025-07-17T08:16:40.564Z
6688a8f30bf195d6e53ac28d
https://cdn-avatars.huggingface.co/v1/production/uploads/6688a8f30bf195d6e53ac28d/5izbVmdCWWwA1wBjcxZPB.jpeg
Yuxi Xiao
user
Yuxihenry
64dd45b3c9a04fc4a5f6638d
Qingyan Bai
64dd45b3c9a04fc4a5f6638e
Juntao Zhang
64dd45b3c9a04fc4a5f6638f
Kecheng Zheng
admin_assigned
2023-08-20T14:13:57.205Z
64252045a4f3051f54dd1d53
/avatars/0e423a3291091be3b4736a14da3ce495.svg
kecheng zheng
user
zkcys001
64dd45b3c9a04fc4a5f66390
Xiaowei Zhou
64dd45b3c9a04fc4a5f66391
Qifeng Chen
admin_assigned
2023-08-20T14:14:48.369Z
6467b121e7a6a374fd19b44b
/avatars/3f2874d58986d651aef55e3408b05700.svg
Qifeng Chen
user
cqf
64dd45b3c9a04fc4a5f66392
Yujun Shen
CoDeF: Content Deformation Fields for Temporally Consistent Video
  Processing
We present the content deformation field CoDeF as a new type of video
representation, which consists of a canonical content field aggregating the
static contents in the entire video and a temporal deformation field recording
the transformations from the canonical image (i.e., rendered from the canonical
content field) to each individual frame along the time axis.Given a target
video, these two fields are jointly optimized to reconstruct it through a
carefully tailored rendering pipeline.We advisedly introduce some
regularizations into the optimization process, urging the canonical content
field to inherit semantics (e.g., the object shape) from the video.With such a
design, CoDeF naturally supports lifting image algorithms for video processing,
in the sense that one can apply an image algorithm to the canonical image and
effortlessly propagate the outcomes to the entire video with the aid of the
temporal deformation field.We experimentally show that CoDeF is able to lift
image-to-image translation to video-to-video translation and lift keypoint
detection to keypoint tracking without any training.More importantly, thanks to
our lifting strategy that deploys the algorithms on only one image, we achieve
superior cross-frame consistency in processed videos compared to existing
video-to-video translation approaches, and even manage to track non-rigid
objects like water and smog.Project page can be found at
https://qiuyu96.github.io/CoDeF/.
CoDeF represents videos using a canonical content field and a temporal deformation field, enabling the lifting of image processing algorithms to video processing with superior cross-frame consistency.
content deformation field canonical content field temporal deformation field rendering pipeline image algorithms video processing image-to-image translation video-to-video translation keypoint detection keypoint tracking cross-frame consistency