Eventful Transformers: Leveraging Temporal Redundancy in Vision
  Transformers
Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method for identifying
and re-processing only those tokens that have changed significantly over time.
Our proposed family of models, Eventful Transformers, can be converted from
existing Transformers (often without any re-training) and give adaptive control
over the compute cost at runtime. We evaluate our method on large-scale
datasets for video object detection (ImageNet VID) and action recognition
(EPIC-Kitchens 100). Our approach leads to significant computational savings
(on the order of 2-4x) with only minor reductions in accuracy.
64ec7b155ff6f83378095381
Matthew Dutson
64ec7b155ff6f83378095382
Yin Li
64ec7b155ff6f83378095383
Mohit Gupta
Eventful Transformers: Leveraging Temporal Redundancy in Vision
  Transformers
Vision Transformers achieve impressive accuracy across a range of visual
recognition tasks. Unfortunately, their accuracy frequently comes with high
computational costs. This is a particular issue in video recognition, where
models are often applied repeatedly across frames or temporal chunks. In this
work, we exploit temporal redundancy between subsequent inputs to reduce the
cost of Transformers for video processing. We describe a method for identifying
and re-processing only those tokens that have changed significantly over time.
Our proposed family of models, Eventful Transformers, can be converted from
existing Transformers (often without any re-training) and give adaptive control
over the compute cost at runtime. We evaluate our method on large-scale
datasets for video object detection (ImageNet VID) and action recognition
(EPIC-Kitchens 100). Our approach leads to significant computational savings
(on the order of 2-4x) with only minor reductions in accuracy.
The proposed Eventful Transformers identify and reprocess only significant changes in video frames to reduce computational costs by 2-4x with minimal accuracy loss.
Vision Transformers computational costs video recognition temporal redundancy tokens Eventful Transformers adaptive control