title: MaskINT: Video Editing via Interpolative Non-autoregressive Masked
    Transformers
summary: Recent advances in generative AI have significantly enhanced image and video
  editing, particularly in the context of text prompt control. State-of-the-art
  approaches predominantly rely on diffusion models to accomplish these tasks.
  However, the computational demands of diffusion-based methods are substantial,
  often necessitating large-scale paired datasets for training, and therefore
  challenging the deployment in practical applications. This study addresses this
  challenge by breaking down the text-based video editing process into two
  separate stages. In the first stage, we leverage an existing text-to-image
  diffusion model to simultaneously edit a few keyframes without additional
  fine-tuning. In the second stage, we introduce an efficient model called
  MaskINT, which is built on non-autoregressive masked generative transformers
  and specializes in frame interpolation between the keyframes, benefiting from
  structural guidance provided by intermediate frames. Our comprehensive set of
  experiments illustrates the efficacy and efficiency of MaskINT when compared to
  other diffusion-based methodologies. This research offers a practical solution
  for text-based video editing and showcases the potential of non-autoregressive
  masked generative transformers in this domain.
paper_authors._id: 6583c1acc73f74776b03d2f5
paper_authors.name: Haoyu Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:36:23.502Z
paper_authors._id: 650a8979c19e5b4c8a6ff062
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/650a8979c19e5b4c8a6ff062/64_JuECX_k_-uK7m7nlua.jpeg
paper_authors.fullname: Haoyu Ma
paper_authors.type: user
paper_authors.user: haoyum1997
paper_authors._id: 6583c1acc73f74776b03d2f6
paper_authors.name: Shahin Mahdizadehaghdam
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-22T07:36:18.457Z
paper_authors._id: 6584b71cb0b0e7e38a1e1b7c
paper_authors.avatarUrl: /avatars/52080f076ddaba4cb11c3779f1e8654f.svg
paper_authors.fullname: Shahin Mahdizadehaghdam
paper_authors.type: user
paper_authors.user: shahinma
paper_authors._id: 6583c1acc73f74776b03d2f7
paper_authors.name: Bichen Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:36:48.563Z
paper_authors._id: 6363749520c2cf37bccd9e84
paper_authors.avatarUrl: /avatars/e9de7911e67090d883f15a7a58f18e4e.svg
paper_authors.fullname: Bichen Wu
paper_authors.type: user
paper_authors.user: bichenwu
paper_authors._id: 6583c1acc73f74776b03d2f8
paper_authors.name: Zhipeng Fan
paper_authors._id: 6583c1acc73f74776b03d2f9
paper_authors.name: Yuchao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:37:38.956Z
paper_authors._id: 63021630a35b21bd8a53305a
paper_authors.avatarUrl: /avatars/7a7e8b39749eda61e57d8a1908726558.svg
paper_authors.fullname: Gu Yuchao
paper_authors.type: user
paper_authors.user: guyuchao
paper_authors._id: 6583c1acc73f74776b03d2fa
paper_authors.name: Wenliang Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:37:48.372Z
paper_authors._id: 6306e962660f01f150a90776
paper_authors.avatarUrl: /avatars/095017b35a5ab0933bbd802d18e96fac.svg
paper_authors.fullname: Wenliang Zhao
paper_authors.type: user
paper_authors.user: wl-zhao
paper_authors._id: 6583c1acc73f74776b03d2fb
paper_authors.name: Lior Shapira
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:37:56.098Z
paper_authors._id: 6350666a0f376d3c482af8e6
paper_authors.avatarUrl: /avatars/bf71d055c669de89a5ad4d6cba6280c0.svg
paper_authors.fullname: Lior Shapira
paper_authors.type: user
paper_authors.user: liorshap77
paper_authors._id: 6583c1acc73f74776b03d2fc
paper_authors.name: Xiaohui Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-21T09:38:03.348Z
paper_authors._id: 64220275ab125f65e85e4c49
paper_authors.avatarUrl: /avatars/fae99aa1abca4d2eb170100bbe462d6e.svg
paper_authors.fullname: Xiaohui Xie
paper_authors.type: user
paper_authors.user: xhx
paper_title: MaskINT: Video Editing via Interpolative Non-autoregressive Masked
    Transformers
paper_summary: Recent advances in generative AI have significantly enhanced image and video
  editing, particularly in the context of text prompt control. State-of-the-art
  approaches predominantly rely on diffusion models to accomplish these tasks.
  However, the computational demands of diffusion-based methods are substantial,
  often necessitating large-scale paired datasets for training, and therefore
  challenging the deployment in practical applications. This study addresses this
  challenge by breaking down the text-based video editing process into two
  separate stages. In the first stage, we leverage an existing text-to-image
  diffusion model to simultaneously edit a few keyframes without additional
  fine-tuning. In the second stage, we introduce an efficient model called
  MaskINT, which is built on non-autoregressive masked generative transformers
  and specializes in frame interpolation between the keyframes, benefiting from
  structural guidance provided by intermediate frames. Our comprehensive set of
  experiments illustrates the efficacy and efficiency of MaskINT when compared to
  other diffusion-based methodologies. This research offers a practical solution
  for text-based video editing and showcases the potential of non-autoregressive
  masked generative transformers in this domain.
paper_ai_keywords: diffusion models text prompt control text-to-image diffusion model fine-tuning MaskINT non-autoregressive masked generative transformers frame interpolation intermediate frames