title: Towards Understanding Unsafe Video Generation
summary: Video generation models (VGMs) have demonstrated the capability to synthesize
  high-quality output. It is important to understand their potential to produce
  unsafe content, such as violent or terrifying videos. In this work, we provide
  a comprehensive understanding of unsafe video generation.
    First, to confirm the possibility that these models could indeed generate
  unsafe videos, we choose unsafe content generation prompts collected from 4chan
  and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After
  filtering out duplicates and poorly generated content, we created an initial
  set of 2112 unsafe videos from an original pool of 5607 videos. Through
  clustering and thematic coding analysis of these generated videos, we identify
  5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic,
  Violent/Bloody, and Political. With IRB approval, we then recruit online
  participants to help label the generated videos. Based on the annotations
  submitted by 403 participants, we identified 937 unsafe videos from the initial
  video set. With the labeled information and the corresponding prompts, we
  created the first dataset of unsafe videos generated by VGMs.
    We then study possible defense mechanisms to prevent the generation of unsafe
  videos. Existing defense methods in image generation focus on filtering either
  input prompt or output results. We propose a new approach called Latent
  Variable Defense (LVD), which works within the model's internal sampling
  process. LVD can achieve 0.90 defense accuracy while reducing time and
  computing resources by 10x when sampling a large number of unsafe prompts.
paper_authors._id: 669895c0414edaf71650f987
paper_authors.name: Yan Pang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:10:42.832Z
paper_authors._id: 63c7523650cc81901dacca85
paper_authors.avatarUrl: /avatars/11676380562aa63361e9bd10532a28e9.svg
paper_authors.fullname: Yan Pang
paper_authors.type: user
paper_authors.user: pypy
paper_authors._id: 669895c0414edaf71650f988
paper_authors.name: Aiping Xiong
paper_authors._id: 669895c0414edaf71650f989
paper_authors.name: Yang Zhang
paper_authors._id: 669895c0414edaf71650f98a
paper_authors.name: Tianhao Wang
paper_title: Towards Understanding Unsafe Video Generation
paper_summary: Video generation models (VGMs) have demonstrated the capability to synthesize
  high-quality output. It is important to understand their potential to produce
  unsafe content, such as violent or terrifying videos. In this work, we provide
  a comprehensive understanding of unsafe video generation.
    First, to confirm the possibility that these models could indeed generate
  unsafe videos, we choose unsafe content generation prompts collected from 4chan
  and Lexica, and three open-source SOTA VGMs to generate unsafe videos. After
  filtering out duplicates and poorly generated content, we created an initial
  set of 2112 unsafe videos from an original pool of 5607 videos. Through
  clustering and thematic coding analysis of these generated videos, we identify
  5 unsafe video categories: Distorted/Weird, Terrifying, Pornographic,
  Violent/Bloody, and Political. With IRB approval, we then recruit online
  participants to help label the generated videos. Based on the annotations
  submitted by 403 participants, we identified 937 unsafe videos from the initial
  video set. With the labeled information and the corresponding prompts, we
  created the first dataset of unsafe videos generated by VGMs.
    We then study possible defense mechanisms to prevent the generation of unsafe
  videos. Existing defense methods in image generation focus on filtering either
  input prompt or output results. We propose a new approach called Latent
  Variable Defense (LVD), which works within the model's internal sampling
  process. LVD can achieve 0.90 defense accuracy while reducing time and
  computing resources by 10x when sampling a large number of unsafe prompts.
paper_ai_summary: The study examines unsafe video generation using advanced video generation models, identifies key categories of unsafe content, and proposes a novel defense mechanism called Latent Variable Defense (LVD) that enhances accuracy and efficiency.
paper_ai_keywords: VGMs unsafe video generation unsafe content generation prompts latent variable defense internal sampling process