title: What Matters in Detecting AI-Generated Videos like Sora?
summary: Recent advancements in diffusion-based video generation have showcased
  remarkable results, yet the gap between synthetic and real-world videos remains
  under-explored. In this study, we examine this gap from three fundamental
  perspectives: appearance, motion, and geometry, comparing real-world videos
  with those generated by a state-of-the-art AI model, Stable Video Diffusion. To
  achieve this, we train three classifiers using 3D convolutional networks, each
  targeting distinct aspects: vision foundation model features for appearance,
  optical flow for motion, and monocular depth for geometry. Each classifier
  exhibits strong performance in fake video detection, both qualitatively and
  quantitatively. This indicates that AI-generated videos are still easily
  detectable, and a significant gap between real and fake videos persists.
  Furthermore, utilizing the Grad-CAM, we pinpoint systematic failures of
  AI-generated videos in appearance, motion, and geometry. Finally, we propose an
  Ensemble-of-Experts model that integrates appearance, optical flow, and depth
  information for fake video detection, resulting in enhanced robustness and
  generalization ability. Our model is capable of detecting videos generated by
  Sora with high accuracy, even without exposure to any Sora videos during
  training. This suggests that the gap between real and fake videos can be
  generalized across various video generative models. Project page:
  https://justin-crchang.github.io/3DCNNDetection.github.io/
paper_authors._id: 66851684fb78936cdbfd16d2
paper_authors.name: Chirui Chang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-03T09:47:54.002Z
paper_authors._id: 64a7fe360f4bad61a44aafbe
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64a7fe360f4bad61a44aafbe/ocs7Qf3AT5q1S3WjGDYDh.jpeg
paper_authors.fullname: Chirui Chang
paper_authors.type: user
paper_authors.user: JustinSheung
paper_authors._id: 66851684fb78936cdbfd16d3
paper_authors.name: Zhengzhe Liu
paper_authors._id: 66851684fb78936cdbfd16d4
paper_authors.name: Xiaoyang Lyu
paper_authors._id: 66851684fb78936cdbfd16d5
paper_authors.name: Xiaojuan Qi
paper_title: What Matters in Detecting AI-Generated Videos like Sora?
paper_summary: Recent advancements in diffusion-based video generation have showcased
  remarkable results, yet the gap between synthetic and real-world videos remains
  under-explored. In this study, we examine this gap from three fundamental
  perspectives: appearance, motion, and geometry, comparing real-world videos
  with those generated by a state-of-the-art AI model, Stable Video Diffusion. To
  achieve this, we train three classifiers using 3D convolutional networks, each
  targeting distinct aspects: vision foundation model features for appearance,
  optical flow for motion, and monocular depth for geometry. Each classifier
  exhibits strong performance in fake video detection, both qualitatively and
  quantitatively. This indicates that AI-generated videos are still easily
  detectable, and a significant gap between real and fake videos persists.
  Furthermore, utilizing the Grad-CAM, we pinpoint systematic failures of
  AI-generated videos in appearance, motion, and geometry. Finally, we propose an
  Ensemble-of-Experts model that integrates appearance, optical flow, and depth
  information for fake video detection, resulting in enhanced robustness and
  generalization ability. Our model is capable of detecting videos generated by
  Sora with high accuracy, even without exposure to any Sora videos during
  training. This suggests that the gap between real and fake videos can be
  generalized across various video generative models. Project page:
  https://justin-crchang.github.io/3DCNNDetection.github.io/
paper_ai_summary: An ensemble of 3D convolutional network classifiers detects fake videos by analyzing appearance, motion, and geometry, revealing persistent gaps between real and synthetic videos.
paper_ai_keywords: diffusion-based video generation 3D convolutional networks vision foundation model features optical flow monocular depth Grad-CAM fake video detection Ensemble-of-Experts model Sora