Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for
  Pre-training and Benchmarks
To promote the development of Vision-Language Pre-training (VLP) and
multimodal Large Language Model (LLM) in the Chinese community, we firstly
release the largest public Chinese high-quality video-language dataset named
Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing
website, with strict criteria of safety, diversity, and quality. Youku-mPLUG
contains 10 million Chinese video-text pairs filtered from 400 million raw
videos across a wide range of 45 diverse categories for large-scale
pre-training. In addition, to facilitate a comprehensive evaluation of
video-language models, we carefully build the largest human-annotated Chinese
benchmarks covering three popular video-language tasks of cross-modal
retrieval, video captioning, and video category classification. Youku-mPLUG can
enable researchers to conduct more in-depth multimodal research and develop
better applications in the future. Furthermore, we release popular
video-language pre-training models, ALPRO and mPLUG-2, and our proposed
modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG.
Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1%
improvement in video category classification. Besides, mPLUG-video achieves a
new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in
video category classification and 68.9 CIDEr score in video captioning,
respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with
only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate
impressive instruction and video understanding ability. The zero-shot
instruction understanding experiment indicates that pretraining with
Youku-mPLUG can enhance the ability to comprehend overall and detailed visual
semantics, recognize scene text, and leverage open-domain knowledge.
648131b4c0d3c70316d5696c
Haiyang Xu
claimed_verified
2023-06-09T15:32:47.874Z
645b10e80c73ea27d13f7aca
/avatars/95e565306472a15067440b5b43e07a6f.svg
xuhaiyang
user
xhyandwyy
648131b4c0d3c70316d5696d
Qinghao Ye
claimed_verified
2023-06-16T07:05:02.550Z
64530fc01a57e1179c1fe4c0
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
QinghaoYe
user
MAGAer13
648131b4c0d3c70316d5696e
Xuan Wu
648131b4c0d3c70316d5696f
Ming Yan
claimed_verified
2023-06-08T16:07:39.345Z
64771cfdd7cf39f2e9381aa9
/avatars/48adf00c3b653df02628f80511639e19.svg
Ming
user
MingYan123
648131b4c0d3c70316d56970
Yuan Miao
648131b4c0d3c70316d56971
Jiabo Ye
648131b4c0d3c70316d56972
Guohai Xu
admin_assigned
2023-06-12T13:56:42.865Z
633fb4bf50d83f5065d5eaa4
/avatars/1cfab32907268bc36b7812558f44a65b.svg
Guohai Xu
user
guohai
648131b4c0d3c70316d56973
Anwen Hu
648131b4c0d3c70316d56974
Yaya Shi
admin_assigned
2023-06-12T13:57:43.965Z
63172db0106740aaad4f9954
/avatars/50f3056c4ba45eed3bbdaced0e1c360f.svg
Yaya Shi
user
ShiYaya
648131b4c0d3c70316d56975
Guangwei Xu
648131b4c0d3c70316d56976
Chenliang Li
648131b4c0d3c70316d56977
Qi Qian
648131b4c0d3c70316d56978
Maofei Que
648131b4c0d3c70316d56979
Ji Zhang
648131b4c0d3c70316d5697a
Xiao Zeng
648131b4c0d3c70316d5697b
Fei Huang
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for
  Pre-training and Benchmarks
To promote the development of Vision-Language Pre-training (VLP) and
multimodal Large Language Model (LLM) in the Chinese community, we firstly
release the largest public Chinese high-quality video-language dataset named
Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing
website, with strict criteria of safety, diversity, and quality. Youku-mPLUG
contains 10 million Chinese video-text pairs filtered from 400 million raw
videos across a wide range of 45 diverse categories for large-scale
pre-training. In addition, to facilitate a comprehensive evaluation of
video-language models, we carefully build the largest human-annotated Chinese
benchmarks covering three popular video-language tasks of cross-modal
retrieval, video captioning, and video category classification. Youku-mPLUG can
enable researchers to conduct more in-depth multimodal research and develop
better applications in the future. Furthermore, we release popular
video-language pre-training models, ALPRO and mPLUG-2, and our proposed
modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG.
Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1%
improvement in video category classification. Besides, mPLUG-video achieves a
new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in
video category classification and 68.9 CIDEr score in video captioning,
respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with
only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate
impressive instruction and video understanding ability. The zero-shot
instruction understanding experiment indicates that pretraining with
Youku-mPLUG can enhance the ability to comprehend overall and detailed visual
semantics, recognize scene text, and leverage open-domain knowledge.
A new large-scale Chinese video-language dataset (Youku-mPLUG) is released, enabling deep pre-training and evaluation of multimodal models. Researchers demonstrated significant improvements in video category classification and captioning using models pre-trained on this dataset.
Vision-Language Pre-training (VLP) multimodal Large Language Model (LLM) Youku-mPLUG video-language dataset video-text pairs cross-modal retrieval video captioning video category classification ALPRO mPLUG-2 modularized decoder-only model pre-trained models zero-shot instruction understanding visual semantics open-domain knowledge