title: OmniCreator: Self-Supervised Unified Generation with Universal Editing
summary: We introduce OmniCreator, a novel framework that can conduct text-prompted
  unified (image+video) generation as well as editing all in one place.
  OmniCreator acquires generative and universal editing capabilities in a
  self-supervised manner, taking original text-video pairs as conditions while
  utilizing the same video as a denoising target to learn the semantic
  correspondence between video and text. During inference, when presented with a
  text prompt and a video, OmniCreator is capable of generating a target that is
  faithful to both, achieving a universal editing effect that is unconstrained as
  opposed to existing editing work that primarily focuses on certain editing
  types or relies on additional controls (e.g., structural conditions, attention
  features, or DDIM inversion). On the other hand, when presented with a text
  prompt only, OmniCreator becomes generative, producing high-quality video as a
  result of the semantic correspondence learned. Importantly, we found that the
  same capabilities extend to images as is, making OmniCreator a truly unified
  framework. Further, due to the lack of existing generative video editing
  benchmarks, we introduce the OmniBench-99 dataset, designed to evaluate the
  performance of generative video editing models comprehensively. Extensive
  experiments demonstrate that OmniCreator exhibits substantial superiority over
  all other models.
paper_authors._id: 674fe64737c700d35214dda6
paper_authors.name: Haodong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:44:20.090Z
paper_authors._id: 6570450a78d7aca0c361a177
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6570450a78d7aca0c361a177/z0GrnXEsjK2_G-hFfQhKv.jpeg
paper_authors.fullname: Harold Chen
paper_authors.type: user
paper_authors.user: Harold328
paper_authors._id: 674fe64737c700d35214dda7
paper_authors.name: Lan Wang
paper_authors._id: 674fe64737c700d35214dda8
paper_authors.name: Harry Yang
paper_authors._id: 674fe64737c700d35214dda9
paper_authors.name: Ser-Nam Lim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-04T12:00:17.400Z
paper_authors._id: 65884498ce38d143c435d279
paper_authors.avatarUrl: /avatars/96a917c40680a79a37dcb8fa75014c21.svg
paper_authors.fullname: Ser Nam Lim
paper_authors.type: user
paper_authors.user: sernam
paper_title: OmniCreator: Self-Supervised Unified Generation with Universal Editing
paper_summary: We introduce OmniCreator, a novel framework that can conduct text-prompted
  unified (image+video) generation as well as editing all in one place.
  OmniCreator acquires generative and universal editing capabilities in a
  self-supervised manner, taking original text-video pairs as conditions while
  utilizing the same video as a denoising target to learn the semantic
  correspondence between video and text. During inference, when presented with a
  text prompt and a video, OmniCreator is capable of generating a target that is
  faithful to both, achieving a universal editing effect that is unconstrained as
  opposed to existing editing work that primarily focuses on certain editing
  types or relies on additional controls (e.g., structural conditions, attention
  features, or DDIM inversion). On the other hand, when presented with a text
  prompt only, OmniCreator becomes generative, producing high-quality video as a
  result of the semantic correspondence learned. Importantly, we found that the
  same capabilities extend to images as is, making OmniCreator a truly unified
  framework. Further, due to the lack of existing generative video editing
  benchmarks, we introduce the OmniBench-99 dataset, designed to evaluate the
  performance of generative video editing models comprehensively. Extensive
  experiments demonstrate that OmniCreator exhibits substantial superiority over
  all other models.
paper_ai_keywords: OmniCreator text-prompted unified generation generative capabilities universal editing capabilities self-supervised learning text-video pairs denoising target semantic correspondence inference universal editing unconstrained editing high-quality video OmniBench-99 dataset generative video editing models