title: Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for
    Pre-training and Benchmarks
summary: To promote the development of Vision-Language Pre-training (VLP) and
  multimodal Large Language Model (LLM) in the Chinese community, we firstly
  release the largest public Chinese high-quality video-language dataset named
  Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing
  website, with strict criteria of safety, diversity, and quality. Youku-mPLUG
  contains 10 million Chinese video-text pairs filtered from 400 million raw
  videos across a wide range of 45 diverse categories for large-scale
  pre-training. In addition, to facilitate a comprehensive evaluation of
  video-language models, we carefully build the largest human-annotated Chinese
  benchmarks covering three popular video-language tasks of cross-modal
  retrieval, video captioning, and video category classification. Youku-mPLUG can
  enable researchers to conduct more in-depth multimodal research and develop
  better applications in the future. Furthermore, we release popular
  video-language pre-training models, ALPRO and mPLUG-2, and our proposed
  modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG.
  Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1%
  improvement in video category classification. Besides, mPLUG-video achieves a
  new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in
  video category classification and 68.9 CIDEr score in video captioning,
  respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with
  only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate
  impressive instruction and video understanding ability. The zero-shot
  instruction understanding experiment indicates that pretraining with
  Youku-mPLUG can enhance the ability to comprehend overall and detailed visual
  semantics, recognize scene text, and leverage open-domain knowledge.
paper_authors._id: 648131b4c0d3c70316d5696c
paper_authors.name: Haiyang Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-09T15:32:47.874Z
paper_authors._id: 645b10e80c73ea27d13f7aca
paper_authors.avatarUrl: /avatars/95e565306472a15067440b5b43e07a6f.svg
paper_authors.fullname: xuhaiyang
paper_authors.type: user
paper_authors.user: xhyandwyy
paper_authors._id: 648131b4c0d3c70316d5696d
paper_authors.name: Qinghao Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-16T07:05:02.550Z
paper_authors._id: 64530fc01a57e1179c1fe4c0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/0lncTpIHXn6suB0p-oSma.jpeg
paper_authors.fullname: QinghaoYe
paper_authors.type: user
paper_authors.user: MAGAer13
paper_authors._id: 648131b4c0d3c70316d5696e
paper_authors.name: Xuan Wu
paper_authors._id: 648131b4c0d3c70316d5696f
paper_authors.name: Ming Yan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:07:39.345Z
paper_authors._id: 64771cfdd7cf39f2e9381aa9
paper_authors.avatarUrl: /avatars/48adf00c3b653df02628f80511639e19.svg
paper_authors.fullname: Ming
paper_authors.type: user
paper_authors.user: MingYan123
paper_authors._id: 648131b4c0d3c70316d56970
paper_authors.name: Yuan Miao
paper_authors._id: 648131b4c0d3c70316d56971
paper_authors.name: Jiabo Ye
paper_authors._id: 648131b4c0d3c70316d56972
paper_authors.name: Guohai Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:56:42.865Z
paper_authors._id: 633fb4bf50d83f5065d5eaa4
paper_authors.avatarUrl: /avatars/1cfab32907268bc36b7812558f44a65b.svg
paper_authors.fullname: Guohai Xu
paper_authors.type: user
paper_authors.user: guohai
paper_authors._id: 648131b4c0d3c70316d56973
paper_authors.name: Anwen Hu
paper_authors._id: 648131b4c0d3c70316d56974
paper_authors.name: Yaya Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:57:43.965Z
paper_authors._id: 63172db0106740aaad4f9954
paper_authors.avatarUrl: /avatars/50f3056c4ba45eed3bbdaced0e1c360f.svg
paper_authors.fullname: Yaya Shi
paper_authors.type: user
paper_authors.user: ShiYaya
paper_authors._id: 648131b4c0d3c70316d56975
paper_authors.name: Guangwei Xu
paper_authors._id: 648131b4c0d3c70316d56976
paper_authors.name: Chenliang Li
paper_authors._id: 648131b4c0d3c70316d56977
paper_authors.name: Qi Qian
paper_authors._id: 648131b4c0d3c70316d56978
paper_authors.name: Maofei Que
paper_authors._id: 648131b4c0d3c70316d56979
paper_authors.name: Ji Zhang
paper_authors._id: 648131b4c0d3c70316d5697a
paper_authors.name: Xiao Zeng
paper_authors._id: 648131b4c0d3c70316d5697b
paper_authors.name: Fei Huang
paper_title: Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for
    Pre-training and Benchmarks
paper_summary: To promote the development of Vision-Language Pre-training (VLP) and
  multimodal Large Language Model (LLM) in the Chinese community, we firstly
  release the largest public Chinese high-quality video-language dataset named
  Youku-mPLUG, which is collected from Youku, a well-known Chinese video-sharing
  website, with strict criteria of safety, diversity, and quality. Youku-mPLUG
  contains 10 million Chinese video-text pairs filtered from 400 million raw
  videos across a wide range of 45 diverse categories for large-scale
  pre-training. In addition, to facilitate a comprehensive evaluation of
  video-language models, we carefully build the largest human-annotated Chinese
  benchmarks covering three popular video-language tasks of cross-modal
  retrieval, video captioning, and video category classification. Youku-mPLUG can
  enable researchers to conduct more in-depth multimodal research and develop
  better applications in the future. Furthermore, we release popular
  video-language pre-training models, ALPRO and mPLUG-2, and our proposed
  modularized decoder-only model mPLUG-video pre-trained on Youku-mPLUG.
  Experiments show that models pre-trained on Youku-mPLUG gain up to 23.1%
  improvement in video category classification. Besides, mPLUG-video achieves a
  new state-of-the-art result on these benchmarks with 80.5% top-1 accuracy in
  video category classification and 68.9 CIDEr score in video captioning,
  respectively. Finally, we scale up mPLUG-video based on the frozen Bloomz with
  only 1.7% trainable parameters as Chinese multimodal LLM, and demonstrate
  impressive instruction and video understanding ability. The zero-shot
  instruction understanding experiment indicates that pretraining with
  Youku-mPLUG can enhance the ability to comprehend overall and detailed visual
  semantics, recognize scene text, and leverage open-domain knowledge.
paper_ai_summary: A new large-scale Chinese video-language dataset (Youku-mPLUG) is released, enabling deep pre-training and evaluation of multimodal models. Researchers demonstrated significant improvements in video category classification and captioning using models pre-trained on this dataset.
paper_ai_keywords: Vision-Language Pre-training (VLP) multimodal Large Language Model (LLM) Youku-mPLUG video-language dataset video-text pairs cross-modal retrieval video captioning video category classification ALPRO mPLUG-2 modularized decoder-only model pre-trained models zero-shot instruction understanding visual semantics open-domain knowledge