title: RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame
    Animated Sticker Generation
summary: Recently, great progress has been made in video generation technology,
  attracting the widespread attention of scholars. To apply this technology to
  downstream applications under resource-constrained conditions, researchers
  usually fine-tune the pre-trained models based on parameter-efficient tuning
  methods such as Adapter or Lora. Although these methods can transfer the
  knowledge from the source domain to the target domain, fewer training
  parameters lead to poor fitting ability, and the knowledge from the source
  domain may lead to the inference process deviating from the target domain. In
  this paper, we argue that under constrained resources, training a smaller video
  generation model from scratch using only million-level samples can outperform
  parameter-efficient tuning on larger models in downstream applications: the
  core lies in the effective utilization of data and curriculum strategy. Take
  animated sticker generation (ASG) as a case study, we first construct a
  discrete frame generation network for stickers with low frame rates, ensuring
  that its parameters meet the requirements of model training under constrained
  resources. In order to provide data support for models trained from scratch, we
  come up with a dual-mask based data utilization strategy, which manages to
  improve the availability and expand the diversity of limited data. To
  facilitate convergence under dual-mask situation, we propose a
  difficulty-adaptive curriculum learning method, which decomposes the sample
  entropy into static and adaptive components so as to obtain samples from easy
  to difficult. The experiment demonstrates that our resource-efficient dual-mask
  training framework is quantitatively and qualitatively superior to
  efficient-parameter tuning methods such as I2V-Adapter and SimDA, verifying the
  feasibility of our method on downstream tasks under constrained resources. Code
  will be available.
paper_authors._id: 67e22bc349edf14060e5747a
paper_authors.name: Zhiqiang Yuan
paper_authors._id: 67e22bc349edf14060e5747b
paper_authors.name: Ting Zhang
paper_authors._id: 67e22bc349edf14060e5747c
paper_authors.name: Ying Deng
paper_authors._id: 67e22bc349edf14060e5747d
paper_authors.name: Jiapei Zhang
paper_authors._id: 67e22bc349edf14060e5747e
paper_authors.name: Yeshuang Zhu
paper_authors._id: 67e22bc349edf14060e5747f
paper_authors.name: Zexi Jia
paper_authors._id: 67e22bc349edf14060e57480
paper_authors.name: Jie Zhou
paper_authors._id: 67e22bc349edf14060e57481
paper_authors.name: Jinchao Zhang
paper_title: RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame
    Animated Sticker Generation
paper_summary: Recently, great progress has been made in video generation technology,
  attracting the widespread attention of scholars. To apply this technology to
  downstream applications under resource-constrained conditions, researchers
  usually fine-tune the pre-trained models based on parameter-efficient tuning
  methods such as Adapter or Lora. Although these methods can transfer the
  knowledge from the source domain to the target domain, fewer training
  parameters lead to poor fitting ability, and the knowledge from the source
  domain may lead to the inference process deviating from the target domain. In
  this paper, we argue that under constrained resources, training a smaller video
  generation model from scratch using only million-level samples can outperform
  parameter-efficient tuning on larger models in downstream applications: the
  core lies in the effective utilization of data and curriculum strategy. Take
  animated sticker generation (ASG) as a case study, we first construct a
  discrete frame generation network for stickers with low frame rates, ensuring
  that its parameters meet the requirements of model training under constrained
  resources. In order to provide data support for models trained from scratch, we
  come up with a dual-mask based data utilization strategy, which manages to
  improve the availability and expand the diversity of limited data. To
  facilitate convergence under dual-mask situation, we propose a
  difficulty-adaptive curriculum learning method, which decomposes the sample
  entropy into static and adaptive components so as to obtain samples from easy
  to difficult. The experiment demonstrates that our resource-efficient dual-mask
  training framework is quantitatively and qualitatively superior to
  efficient-parameter tuning methods such as I2V-Adapter and SimDA, verifying the
  feasibility of our method on downstream tasks under constrained resources. Code
  will be available.
paper_ai_summary: Training smaller video generation models from scratch with a dual-mask data utilization strategy and difficulty-adaptive curriculum learning outperforms parameter-efficient tuning methods under resource constraints.
paper_ai_keywords: Adapter Lora parameter-efficient tuning discrete frame generation dual-mask curriculum learning sample entropy