title: Optimizing ViViT Training: Time and Memory Reduction for Action
    Recognition
summary: In this paper, we address the challenges posed by the substantial training
  time and memory consumption associated with video transformers, focusing on the
  ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
  version, as our baseline for action recognition tasks. The factorised encoder
  variant follows the late-fusion approach that is adopted by many state of the
  art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
  among the different variants of ViViT, its considerable training time and
  memory requirements still pose a significant barrier to entry. Our method is
  designed to lower this barrier and is based on the idea of freezing the spatial
  transformer during training. This leads to a low accuracy model if naively
  done. But we show that by (1) appropriately initializing the temporal
  transformer (a module responsible for processing temporal information) (2)
  introducing a compact adapter model connecting frozen spatial representations
  ((a module that selectively focuses on regions of the input image) to the
  temporal transformer, we can enjoy the benefits of freezing the spatial
  transformer without sacrificing accuracy. Through extensive experimentation
  over 6 benchmarks, we demonstrate that our proposed training strategy
  significantly reduces training costs (by sim 50%) and memory consumption
  while maintaining or slightly improving performance by up to 1.79\% compared to
  the baseline model. Our approach additionally unlocks the capability to utilize
  larger image transformer models as our spatial transformer and access more
  frames with the same memory consumption.
paper_authors._id: 6482b89556e161f11f28014b
paper_authors.name: Shreyank N Gowda
paper_authors._id: 6482b89556e161f11f28014c
paper_authors.name: Anurag Arnab
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T13:16:36.095Z
paper_authors._id: 63d7f0466ac3104e50c5311e
paper_authors.avatarUrl: /avatars/2421bfb7747664d1acefbac1c00c3475.svg
paper_authors.fullname: Anurag Arnab
paper_authors.type: user
paper_authors.user: anurag-arnab
paper_authors._id: 6482b89556e161f11f28014d
paper_authors.name: Jonathan Huang
paper_title: Optimizing ViViT Training: Time and Memory Reduction for Action
    Recognition
paper_summary: In this paper, we address the challenges posed by the substantial training
  time and memory consumption associated with video transformers, focusing on the
  ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
  version, as our baseline for action recognition tasks. The factorised encoder
  variant follows the late-fusion approach that is adopted by many state of the
  art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
  among the different variants of ViViT, its considerable training time and
  memory requirements still pose a significant barrier to entry. Our method is
  designed to lower this barrier and is based on the idea of freezing the spatial
  transformer during training. This leads to a low accuracy model if naively
  done. But we show that by (1) appropriately initializing the temporal
  transformer (a module responsible for processing temporal information) (2)
  introducing a compact adapter model connecting frozen spatial representations
  ((a module that selectively focuses on regions of the input image) to the
  temporal transformer, we can enjoy the benefits of freezing the spatial
  transformer without sacrificing accuracy. Through extensive experimentation
  over 6 benchmarks, we demonstrate that our proposed training strategy
  significantly reduces training costs (by sim 50%) and memory consumption
  while maintaining or slightly improving performance by up to 1.79\% compared to
  the baseline model. Our approach additionally unlocks the capability to utilize
  larger image transformer models as our spatial transformer and access more
  frames with the same memory consumption.
paper_ai_keywords: Video Vision Transformer (ViViT) factorised encoder late-fusion spatial transformer temporal transformer compact adapter model training strategy performance