title: COSMO: COntrastive Streamlined MultimOdal Model with Interleaved
    Pre-Training
summary: In the evolution of Vision-Language Pre-training, shifting from short-text
  comprehension to encompassing extended textual contexts is pivotal. Recent
  autoregressive vision-language models like flamingo, palme, leveraging
  the long-context capability of Large Language Models, have excelled in few-shot
  text generation tasks but face challenges in alignment tasks. Addressing this
  gap, we introduce the contrastive loss into text generation models, presenting
  the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically
  partitioning the language model into dedicated unimodal text processing and
  adept multimodal data handling components. \ModelName, our unified framework,
  merges unimodal and multimodal elements, enhancing model performance for tasks
  involving textual and visual data while notably reducing learnable parameters.
  However, these models demand extensive long-text datasets, yet the availability
  of high-quality long-text video datasets remains limited. To bridge this gap,
  this work introduces \VideoDatasetName, an inaugural interleaved video-text
  dataset featuring comprehensive captions, marking a significant step forward.
  Demonstrating its impact, we illustrate how  enhances model
  performance in image-text tasks. With 34% learnable parameters and utilizing
  72\% of the available data, our model demonstrates significant superiority over
  OpenFlamingo~openflamingo. For instance, in the 4-shot flickr captioning
  task, performance notably improves from 57.2% to 65.\%. The contributions of
   and  are underscored by notable performance
  gains across 14 diverse downstream datasets encompassing both image-text and
  video-text tasks.
paper_authors._id: 6593966b3533188375b72db1
paper_authors.name: Alex Jinpeng Wang
paper_authors._id: 6593966b3533188375b72db2
paper_authors.name: Linjie Li
paper_authors._id: 6593966b3533188375b72db3
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:34:33.745Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 6593966b3533188375b72db4
paper_authors.name: Jianfeng Wang
paper_authors._id: 6593966b3533188375b72db5
paper_authors.name: Kevin Lin
paper_authors._id: 6593966b3533188375b72db6
paper_authors.name: Zhengyuan Yang
paper_authors._id: 6593966b3533188375b72db7
paper_authors.name: Lijuan Wang
paper_authors._id: 6593966b3533188375b72db8
paper_authors.name: Mike Zheng Shou
paper_title: COSMO: COntrastive Streamlined MultimOdal Model with Interleaved
    Pre-Training
paper_summary: In the evolution of Vision-Language Pre-training, shifting from short-text
  comprehension to encompassing extended textual contexts is pivotal. Recent
  autoregressive vision-language models like flamingo, palme, leveraging
  the long-context capability of Large Language Models, have excelled in few-shot
  text generation tasks but face challenges in alignment tasks. Addressing this
  gap, we introduce the contrastive loss into text generation models, presenting
  the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically
  partitioning the language model into dedicated unimodal text processing and
  adept multimodal data handling components. \ModelName, our unified framework,
  merges unimodal and multimodal elements, enhancing model performance for tasks
  involving textual and visual data while notably reducing learnable parameters.
  However, these models demand extensive long-text datasets, yet the availability
  of high-quality long-text video datasets remains limited. To bridge this gap,
  this work introduces \VideoDatasetName, an inaugural interleaved video-text
  dataset featuring comprehensive captions, marking a significant step forward.
  Demonstrating its impact, we illustrate how  enhances model
  performance in image-text tasks. With 34% learnable parameters and utilizing
  72\% of the available data, our model demonstrates significant superiority over
  OpenFlamingo~openflamingo. For instance, in the 4-shot flickr captioning
  task, performance notably improves from 57.2% to 65.\%. The contributions of
   and  are underscored by notable performance
  gains across 14 diverse downstream datasets encompassing both image-text and
  video-text tasks.
paper_ai_summary: A contrastive loss and unified multimodal framework improve vision-language pre-training models, particularly in tasks involving long textual contexts and extended video data, outperforming existing models and utilizing a novel interleaved video-text dataset.
paper_ai_keywords: autoregressive vision-language models contrastive loss multimodal framework unimodal text processing multimodal data handling interleaved video-text dataset learned parameters flickr captioning task