RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame
  Animated Sticker Generation
Recently, great progress has been made in video generation technology,
attracting the widespread attention of scholars. To apply this technology to
downstream applications under resource-constrained conditions, researchers
usually fine-tune the pre-trained models based on parameter-efficient tuning
methods such as Adapter or Lora. Although these methods can transfer the
knowledge from the source domain to the target domain, fewer training
parameters lead to poor fitting ability, and the knowledge from the source
domain may lead to the inference process deviating from the target domain. In
this paper, we argue that under constrained resources, training a smaller video
generation model from scratch using only million-level samples can outperform
parameter-efficient tuning on larger models in downstream applications: the
core lies in the effective utilization of data and curriculum strategy. Take
animated sticker generation (ASG) as a case study, we first construct a
discrete frame generation network for stickers with low frame rates, ensuring
that its parameters meet the requirements of model training under constrained
resources. In order to provide data support for models trained from scratch, we
come up with a dual-mask based data utilization strategy, which manages to
improve the availability and expand the diversity of limited data. To
facilitate convergence under dual-mask situation, we propose a
difficulty-adaptive curriculum learning method, which decomposes the sample
entropy into static and adaptive components so as to obtain samples from easy
to difficult. The experiment demonstrates that our resource-efficient dual-mask
training framework is quantitatively and qualitatively superior to
efficient-parameter tuning methods such as I2V-Adapter and SimDA, verifying the
feasibility of our method on downstream tasks under constrained resources. Code
will be available.
67e22bc349edf14060e5747a
Zhiqiang Yuan
67e22bc349edf14060e5747b
Ting Zhang
67e22bc349edf14060e5747c
Ying Deng
67e22bc349edf14060e5747d
Jiapei Zhang
67e22bc349edf14060e5747e
Yeshuang Zhu
67e22bc349edf14060e5747f
Zexi Jia
67e22bc349edf14060e57480
Jie Zhou
67e22bc349edf14060e57481
Jinchao Zhang
RDTF: Resource-efficient Dual-mask Training Framework for Multi-frame
  Animated Sticker Generation
Recently, great progress has been made in video generation technology,
attracting the widespread attention of scholars. To apply this technology to
downstream applications under resource-constrained conditions, researchers
usually fine-tune the pre-trained models based on parameter-efficient tuning
methods such as Adapter or Lora. Although these methods can transfer the
knowledge from the source domain to the target domain, fewer training
parameters lead to poor fitting ability, and the knowledge from the source
domain may lead to the inference process deviating from the target domain. In
this paper, we argue that under constrained resources, training a smaller video
generation model from scratch using only million-level samples can outperform
parameter-efficient tuning on larger models in downstream applications: the
core lies in the effective utilization of data and curriculum strategy. Take
animated sticker generation (ASG) as a case study, we first construct a
discrete frame generation network for stickers with low frame rates, ensuring
that its parameters meet the requirements of model training under constrained
resources. In order to provide data support for models trained from scratch, we
come up with a dual-mask based data utilization strategy, which manages to
improve the availability and expand the diversity of limited data. To
facilitate convergence under dual-mask situation, we propose a
difficulty-adaptive curriculum learning method, which decomposes the sample
entropy into static and adaptive components so as to obtain samples from easy
to difficult. The experiment demonstrates that our resource-efficient dual-mask
training framework is quantitatively and qualitatively superior to
efficient-parameter tuning methods such as I2V-Adapter and SimDA, verifying the
feasibility of our method on downstream tasks under constrained resources. Code
will be available.
Training smaller video generation models from scratch with a dual-mask data utilization strategy and difficulty-adaptive curriculum learning outperforms parameter-efficient tuning methods under resource constraints.
Adapter Lora parameter-efficient tuning discrete frame generation dual-mask curriculum learning sample entropy