title: Searching Priors Makes Text-to-Video Synthesis Better
summary: Significant advancements in video diffusion models have brought substantial
  progress to the field of text-to-video (T2V) synthesis. However, existing T2V
  synthesis model struggle to accurately generate complex motion dynamics,
  leading to a reduction in video realism. One possible solution is to collect
  massive data and train the model on it, but this would be extremely expensive.
  To alleviate this problem, in this paper, we reformulate the typical T2V
  generation process as a search-based generation pipeline. Instead of scaling up
  the model training, we employ existing videos as the motion prior database.
  Specifically, we divide T2V generation process into two steps: (i) For a given
  prompt input, we search existing text-video datasets to find videos with text
  labels that closely match the prompt motions. We propose a tailored search
  algorithm that emphasizes object motion features. (ii) Retrieved videos are
  processed and distilled into motion priors to fine-tune a pre-trained base T2V
  model, followed by generating desired videos using input prompt. By utilizing
  the priors gleaned from the searched videos, we enhance the realism of the
  generated videos' motion. All operations can be finished on a single NVIDIA RTX
  4090 GPU. We validate our method against state-of-the-art T2V models across
  diverse prompt inputs. The code will be public.
paper_authors._id: 66613546754f90a8582cfbfa
paper_authors.name: Haoran Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:05:26.459Z
paper_authors._id: 652cd5ee66313ebb61cac09c
paper_authors.avatarUrl: /avatars/fcbae1dc7134e625f2d79f767b34e142.svg
paper_authors.fullname: Haoran Cheng
paper_authors.type: user
paper_authors.user: chrxx98
paper_authors._id: 66613546754f90a8582cfbfb
paper_authors.name: Liang Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-06T10:53:31.404Z
paper_authors._id: 66618e365fd9d736e68426d1
paper_authors.avatarUrl: /avatars/76cb9044866017749c626001da1762e8.svg
paper_authors.fullname: PengLiang
paper_authors.type: user
paper_authors.user: PengLiang002
paper_authors._id: 66613546754f90a8582cfbfc
paper_authors.name: Linxuan Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:05:57.120Z
paper_authors._id: 6459e93cb8e78d46d7e046b5
paper_authors.avatarUrl: /avatars/8c944d27f934760fecd31942464ea9de.svg
paper_authors.fullname: Linxuan Xia
paper_authors.type: user
paper_authors.user: Charlo7te
paper_authors._id: 66613546754f90a8582cfbfd
paper_authors.name: Yuepeng Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:06:04.579Z
paper_authors._id: 65ce2c9e3f1b48716ea36271
paper_authors.avatarUrl: /avatars/cb238ed8549150f88207d9206787d083.svg
paper_authors.fullname: Yuepeng Hu
paper_authors.type: user
paper_authors.user: HifiHu
paper_authors._id: 66613546754f90a8582cfbfe
paper_authors.name: Hengjia Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-04T09:47:22.910Z
paper_authors._id: 64631cbd39359568c63a7fd6
paper_authors.avatarUrl: /avatars/e921b6a25314bf14292ffdd0bc6f37de.svg
paper_authors.fullname: Li
paper_authors.type: user
paper_authors.user: Pluto5
paper_authors._id: 66613546754f90a8582cfbff
paper_authors.name: Qinglin Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T10:07:00.863Z
paper_authors._id: 64c13276b4fd9b97756b1783
paper_authors.avatarUrl: /avatars/a1d0480fa94f59ed5db3a1bba09bd660.svg
paper_authors.fullname: lu
paper_authors.type: user
paper_authors.user: qinglin3
paper_authors._id: 66613546754f90a8582cfc00
paper_authors.name: Xiaofei He
paper_authors._id: 66613546754f90a8582cfc01
paper_authors.name: Boxi Wu
paper_title: Searching Priors Makes Text-to-Video Synthesis Better
paper_summary: Significant advancements in video diffusion models have brought substantial
  progress to the field of text-to-video (T2V) synthesis. However, existing T2V
  synthesis model struggle to accurately generate complex motion dynamics,
  leading to a reduction in video realism. One possible solution is to collect
  massive data and train the model on it, but this would be extremely expensive.
  To alleviate this problem, in this paper, we reformulate the typical T2V
  generation process as a search-based generation pipeline. Instead of scaling up
  the model training, we employ existing videos as the motion prior database.
  Specifically, we divide T2V generation process into two steps: (i) For a given
  prompt input, we search existing text-video datasets to find videos with text
  labels that closely match the prompt motions. We propose a tailored search
  algorithm that emphasizes object motion features. (ii) Retrieved videos are
  processed and distilled into motion priors to fine-tune a pre-trained base T2V
  model, followed by generating desired videos using input prompt. By utilizing
  the priors gleaned from the searched videos, we enhance the realism of the
  generated videos' motion. All operations can be finished on a single NVIDIA RTX
  4090 GPU. We validate our method against state-of-the-art T2V models across
  diverse prompt inputs. The code will be public.
paper_ai_keywords: video diffusion models text-to-video (T2V) synthesis complex motion dynamics video realism search-based generation pipeline motion prior database object motion features tailored search algorithm motion priors pre-trained base T2V model