title: EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the
    Backbone
summary: Video-language pre-training (VLP) has become increasingly important due to
  its ability to generalize to various vision and language tasks. However,
  existing egocentric VLP frameworks utilize separate video and language encoders
  and learn task-specific cross-modal information only during fine-tuning,
  limiting the development of a unified system. In this work, we introduce the
  second generation of egocentric video-language pre-training (EgoVLPv2), a
  significant improvement from the previous generation, by incorporating
  cross-modal fusion directly into the video and language backbones. EgoVLPv2
  learns strong video-text representation during pre-training and reuses the
  cross-modal attention modules to support different downstream tasks in a
  flexible and efficient manner, reducing fine-tuning costs. Moreover, our
  proposed fusion in the backbone strategy is more lightweight and
  compute-efficient than stacking additional fusion-specific layers. Extensive
  experiments on a wide range of VL tasks demonstrate the effectiveness of
  EgoVLPv2 by achieving consistent state-of-the-art performance over strong
  baselines across all downstream. Our project page can be found at
  https://shramanpramanick.github.io/EgoVLPv2/.
paper_authors._id: 64ae1a65692be9d12f8ca702
paper_authors.name: Shraman Pramanick
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T08:32:31.322Z
paper_authors._id: 64aec10a9135f79b09117476
paper_authors.avatarUrl: /avatars/88a6bcfeee99c12496940575a2fc65e9.svg
paper_authors.fullname: Shraman Pramanick
paper_authors.type: user
paper_authors.user: Shraman
paper_authors._id: 64ae1a65692be9d12f8ca703
paper_authors.name: Yale Song
paper_authors._id: 64ae1a65692be9d12f8ca704
paper_authors.name: Sayan Nag
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-13T08:32:37.150Z
paper_authors._id: 61de7e35c201b96c5db3bec6
paper_authors.avatarUrl: /avatars/d69c9fcbeff60f6f1c442b2a1226307a.svg
paper_authors.fullname: Sayan Nag
paper_authors.type: user
paper_authors.user: nagsayan
paper_authors._id: 64ae1a65692be9d12f8ca705
paper_authors.name: Kevin Qinghong Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-04T09:33:58.364Z
paper_authors._id: 64440be5af034cdfd69ca3a7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64440be5af034cdfd69ca3a7/qmx24QiDFT29vleCxL9TX.jpeg
paper_authors.fullname: Qinghong (Kevin) Lin
paper_authors.type: user
paper_authors.user: KevinQHLin
paper_authors._id: 64ae1a65692be9d12f8ca706
paper_authors.name: Hardik Shah
paper_authors._id: 64ae1a65692be9d12f8ca707
paper_authors.name: Mike Zheng Shou
paper_authors._id: 64ae1a65692be9d12f8ca708
paper_authors.name: Rama Chellappa
paper_authors._id: 64ae1a65692be9d12f8ca709
paper_authors.name: Pengchuan Zhang
paper_title: EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the
    Backbone
paper_summary: Video-language pre-training (VLP) has become increasingly important due to
  its ability to generalize to various vision and language tasks. However,
  existing egocentric VLP frameworks utilize separate video and language encoders
  and learn task-specific cross-modal information only during fine-tuning,
  limiting the development of a unified system. In this work, we introduce the
  second generation of egocentric video-language pre-training (EgoVLPv2), a
  significant improvement from the previous generation, by incorporating
  cross-modal fusion directly into the video and language backbones. EgoVLPv2
  learns strong video-text representation during pre-training and reuses the
  cross-modal attention modules to support different downstream tasks in a
  flexible and efficient manner, reducing fine-tuning costs. Moreover, our
  proposed fusion in the backbone strategy is more lightweight and
  compute-efficient than stacking additional fusion-specific layers. Extensive
  experiments on a wide range of VL tasks demonstrate the effectiveness of
  EgoVLPv2 by achieving consistent state-of-the-art performance over strong
  baselines across all downstream. Our project page can be found at
  https://shramanpramanick.github.io/EgoVLPv2/.
paper_ai_summary: EgoVLPv2 improves video-language pre-training by integrating cross-modal fusion directly into the backbones, enhancing representation and reducing fine-tuning costs across various tasks.
paper_ai_keywords: egocentric VLP cross-modal fusion video-text representation cross-modal attention modules fine-tuning costs VL tasks