title: VideoCon: Robust Video-Language Alignment via Contrast Captions
summary: Despite being (pre)trained on a massive amount of data, state-of-the-art
  video-language alignment models are not robust to semantically-plausible
  contrastive changes in the video captions. Our work addresses this by
  identifying a broad spectrum of contrast misalignments, such as replacing
  entities, actions, and flipping event order, which alignment models should be
  robust against. To this end, we introduce the VideoCon, a video-language
  alignment dataset constructed by a large language model that generates
  plausible contrast video captions and explanations for differences between
  original and contrast video captions. Then, a generative video-language model
  is finetuned with VideoCon to assess video-language entailment and generate
  explanations. Our VideoCon-based alignment model significantly outperforms
  current models. It exhibits a 12-point increase in AUC for the video-language
  alignment task on human-generated contrast captions. Finally, our model sets
  new state of the art zero-shot performance in temporally-extensive
  video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video
  question answering (ATP-Hard). Moreover, our model shows superior performance
  on novel videos and human-crafted captions and explanations. Our code and data
  are available at https://github.com/Hritikbansal/videocon.
paper_authors._id: 655abc91e0a6202d3682c994
paper_authors.name: Hritik Bansal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T12:39:51.613Z
paper_authors._id: 61c5c25705aa54027c52f7b3
paper_authors.avatarUrl: /avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
paper_authors.fullname: Hritik Bansal
paper_authors.type: user
paper_authors.user: hbXNov
paper_authors._id: 655abc91e0a6202d3682c995
paper_authors.name: Yonatan Bitton
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T12:39:59.082Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 655abc91e0a6202d3682c996
paper_authors.name: Idan Szpektor
paper_authors._id: 655abc91e0a6202d3682c997
paper_authors.name: Kai-Wei Chang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-20T12:40:40.527Z
paper_authors._id: 60b7b9d71b90c5d07c23fbd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1622653364258-noauth.jpeg
paper_authors.fullname: Kai-Wei Chang
paper_authors.type: user
paper_authors.user: kaiweichang
paper_authors._id: 655abc91e0a6202d3682c998
paper_authors.name: Aditya Grover
paper_title: VideoCon: Robust Video-Language Alignment via Contrast Captions
paper_summary: Despite being (pre)trained on a massive amount of data, state-of-the-art
  video-language alignment models are not robust to semantically-plausible
  contrastive changes in the video captions. Our work addresses this by
  identifying a broad spectrum of contrast misalignments, such as replacing
  entities, actions, and flipping event order, which alignment models should be
  robust against. To this end, we introduce the VideoCon, a video-language
  alignment dataset constructed by a large language model that generates
  plausible contrast video captions and explanations for differences between
  original and contrast video captions. Then, a generative video-language model
  is finetuned with VideoCon to assess video-language entailment and generate
  explanations. Our VideoCon-based alignment model significantly outperforms
  current models. It exhibits a 12-point increase in AUC for the video-language
  alignment task on human-generated contrast captions. Finally, our model sets
  new state of the art zero-shot performance in temporally-extensive
  video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video
  question answering (ATP-Hard). Moreover, our model shows superior performance
  on novel videos and human-crafted captions and explanations. Our code and data
  are available at https://github.com/Hritikbansal/videocon.
paper_ai_keywords: speculative contrastive changes contrast misalignments VideoCon video-language-alignment dataset large language model plausible contrast video captions generative video-language model video-language entailment AUC zero-shot performance temporally-extensive video-language tasks text-to-to augmentation video question answering