SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
<|reference_start|>SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset: To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms.<|reference_end|>
@article{dai2024safesora:,
    title={SafeSora: Towards Safety Alignment of Text2Video Generation via a Human
  Preference Dataset},
    author={Josef Dai, Tianle Chen, Xuyao Wang, Ziran Yang, Taiye Chen, Jiaming
  Ji, Yaodong Yang},
    journal={arXiv preprint arXiv:2406.14477},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.14477},
    primaryClass={cs.CV cs.AI cs.DB}
}