Optimizing ViViT Training: Time and Memory Reduction for Action
  Recognition
In this paper, we address the challenges posed by the substantial training
time and memory consumption associated with video transformers, focusing on the
ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
version, as our baseline for action recognition tasks. The factorised encoder
variant follows the late-fusion approach that is adopted by many state of the
art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
among the different variants of ViViT, its considerable training time and
memory requirements still pose a significant barrier to entry. Our method is
designed to lower this barrier and is based on the idea of freezing the spatial
transformer during training. This leads to a low accuracy model if naively
done. But we show that by (1) appropriately initializing the temporal
transformer (a module responsible for processing temporal information) (2)
introducing a compact adapter model connecting frozen spatial representations
((a module that selectively focuses on regions of the input image) to the
temporal transformer, we can enjoy the benefits of freezing the spatial
transformer without sacrificing accuracy. Through extensive experimentation
over 6 benchmarks, we demonstrate that our proposed training strategy
significantly reduces training costs (by sim 50%) and memory consumption
while maintaining or slightly improving performance by up to 1.79\% compared to
the baseline model. Our approach additionally unlocks the capability to utilize
larger image transformer models as our spatial transformer and access more
frames with the same memory consumption.
6482b89556e161f11f28014b
Shreyank N Gowda
6482b89556e161f11f28014c
Anurag Arnab
admin_assigned
2023-06-12T13:16:36.095Z
63d7f0466ac3104e50c5311e
/avatars/2421bfb7747664d1acefbac1c00c3475.svg
Anurag Arnab
user
anurag-arnab
6482b89556e161f11f28014d
Jonathan Huang
Optimizing ViViT Training: Time and Memory Reduction for Action
  Recognition
In this paper, we address the challenges posed by the substantial training
time and memory consumption associated with video transformers, focusing on the
ViViT (Video Vision Transformer) model, in particular the Factorised Encoder
version, as our baseline for action recognition tasks. The factorised encoder
variant follows the late-fusion approach that is adopted by many state of the
art approaches. Despite standing out for its favorable speed/accuracy tradeoffs
among the different variants of ViViT, its considerable training time and
memory requirements still pose a significant barrier to entry. Our method is
designed to lower this barrier and is based on the idea of freezing the spatial
transformer during training. This leads to a low accuracy model if naively
done. But we show that by (1) appropriately initializing the temporal
transformer (a module responsible for processing temporal information) (2)
introducing a compact adapter model connecting frozen spatial representations
((a module that selectively focuses on regions of the input image) to the
temporal transformer, we can enjoy the benefits of freezing the spatial
transformer without sacrificing accuracy. Through extensive experimentation
over 6 benchmarks, we demonstrate that our proposed training strategy
significantly reduces training costs (by sim 50%) and memory consumption
while maintaining or slightly improving performance by up to 1.79\% compared to
the baseline model. Our approach additionally unlocks the capability to utilize
larger image transformer models as our spatial transformer and access more
frames with the same memory consumption.
A training strategy that freezes the spatial transformer in ViViT models reduces training costs and memory consumption while maintaining or improving performance.
ViViT Factorised Encoder late-fusion spatial transformer temporal transformer compact adapter model action recognition