title: SIGMA: Sinkhorn-Guided Masked Video Modeling
summary: Video-based pretraining offers immense potential for learning strong visual
  representations on an unprecedented scale. Recently, masked video modeling
  methods have shown promising scalability, yet fall short in capturing
  higher-level semantics due to reconstructing predefined low-level targets such
  as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling
  (SIGMA), a novel video pretraining method that jointly learns the video model
  in addition to a target feature space using a projection network. However, this
  simple modification means that the regular L2 reconstruction loss will lead to
  trivial solutions as both networks are jointly optimized. As a solution, we
  distribute features of space-time tubes evenly across a limited number of
  learnable clusters. By posing this as an optimal transport problem, we enforce
  high entropy in the generated features across the batch, infusing semantic and
  temporal meaning into the feature space. The resulting cluster assignments are
  used as targets for a symmetric prediction task where the video model predicts
  cluster assignment of the projection network and vice versa. Experimental
  results on ten datasets across three benchmarks validate the effectiveness of
  SIGMA in learning more performant, temporally-aware, and robust video
  representations improving upon state-of-the-art methods. Our project website
  with code is available at: https://quva-lab.github.io/SIGMA.
paper_authors._id: 66a0b2d9684c17d8793c8a7c
paper_authors.name: Mohammadreza Salehi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T08:12:04.687Z
paper_authors._id: 66990ec3aa500cd99d640a19
paper_authors.avatarUrl: /avatars/3520cab2b946158b2cdf6ed84f9516e0.svg
paper_authors.fullname: Mohammadreza Salehi
paper_authors.type: user
paper_authors.user: SMSD75
paper_authors._id: 66a0b2d9684c17d8793c8a7d
paper_authors.name: Michael Dorkenwald
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-24T08:01:01.006Z
paper_authors._id: 6566436e3d526e3297c1b4a1
paper_authors.avatarUrl: /avatars/e433ffccb0f9c2d91649f9988655fedd.svg
paper_authors.fullname: Michael Dorkenwald
paper_authors.type: user
paper_authors.user: mdorkenw
paper_authors._id: 66a0b2d9684c17d8793c8a7e
paper_authors.name: Fida Mohammad Thoker
paper_authors._id: 66a0b2d9684c17d8793c8a7f
paper_authors.name: Efstratios Gavves
paper_authors._id: 66a0b2d9684c17d8793c8a80
paper_authors.name: Cees G. M. Snoek
paper_authors._id: 66a0b2d9684c17d8793c8a81
paper_authors.name: Yuki M. Asano
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-15T09:11:47.423Z
paper_authors._id: 637d21239a5217b88b7549c3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637d21239a5217b88b7549c3/LrIGPiva5VGVZG87rTAJz.jpeg
paper_authors.fullname: Yuki Asano
paper_authors.type: user
paper_authors.user: yukimasano
paper_title: SIGMA: Sinkhorn-Guided Masked Video Modeling
paper_summary: Video-based pretraining offers immense potential for learning strong visual
  representations on an unprecedented scale. Recently, masked video modeling
  methods have shown promising scalability, yet fall short in capturing
  higher-level semantics due to reconstructing predefined low-level targets such
  as pixels. To tackle this, we present Sinkhorn-guided Masked Video Modelling
  (SIGMA), a novel video pretraining method that jointly learns the video model
  in addition to a target feature space using a projection network. However, this
  simple modification means that the regular L2 reconstruction loss will lead to
  trivial solutions as both networks are jointly optimized. As a solution, we
  distribute features of space-time tubes evenly across a limited number of
  learnable clusters. By posing this as an optimal transport problem, we enforce
  high entropy in the generated features across the batch, infusing semantic and
  temporal meaning into the feature space. The resulting cluster assignments are
  used as targets for a symmetric prediction task where the video model predicts
  cluster assignment of the projection network and vice versa. Experimental
  results on ten datasets across three benchmarks validate the effectiveness of
  SIGMA in learning more performant, temporally-aware, and robust video
  representations improving upon state-of-the-art methods. Our project website
  with code is available at: https://quva-lab.github.io/SIGMA.
paper_ai_keywords: masked video modeling Sinkhorn-guided Masked Video Modelling (SIGMA) video pretraining method target feature space projection network L2 reconstruction loss space-time tubes optimal transport problem cluster assignments symmetric prediction task temporally-aware state-of-the-art methods