TRecViT: A Recurrent Video Transformer
We propose a novel block for video modelling. It relies on a
time-space-channel factorisation with dedicated blocks for each dimension:
gated linear recurrent units (LRUs) perform information mixing over time,
self-attention layers perform mixing over space, and MLPs over channels. The
resulting architecture TRecViT performs well on sparse and dense tasks, trained
in supervised or self-supervised regimes. Notably, our model is causal and
outperforms or is on par with a pure attention model ViViT-L on large scale
video datasets (SSv2, Kinetics400), while having 3times less parameters,
12times smaller memory footprint, and 5times lower FLOPs count. Code and
checkpoints will be made available online at
https://github.com/google-deepmind/trecvit.
6769b72361c7635a1e1e5dc0
Viorica Pătrăucean
extracted_confirmed
2025-01-06T10:47:22.333Z
640b53d991d9f65c58a85d4a
/avatars/bea43ffca19cb0705caea3b22f652adc.svg
Vio Patr
user
viorik
6769b72361c7635a1e1e5dc1
Xu Owen He
6769b72361c7635a1e1e5dc2
Joseph Heyward
6769b72361c7635a1e1e5dc3
Chuhan Zhang
6769b72361c7635a1e1e5dc4
Mehdi S. M. Sajjadi
6769b72361c7635a1e1e5dc5
George-Cristian Muraru
6769b72361c7635a1e1e5dc6
Artem Zholus
6769b72361c7635a1e1e5dc7
Mahdi Karami
6769b72361c7635a1e1e5dc8
Ross Goroshin
6769b72361c7635a1e1e5dc9
Yutian Chen
6769b72361c7635a1e1e5dca
Simon Osindero
6769b72361c7635a1e1e5dcb
João Carreira
6769b72361c7635a1e1e5dcc
Razvan Pascanu
TRecViT: A Recurrent Video Transformer
We propose a novel block for video modelling. It relies on a
time-space-channel factorisation with dedicated blocks for each dimension:
gated linear recurrent units (LRUs) perform information mixing over time,
self-attention layers perform mixing over space, and MLPs over channels. The
resulting architecture TRecViT performs well on sparse and dense tasks, trained
in supervised or self-supervised regimes. Notably, our model is causal and
outperforms or is on par with a pure attention model ViViT-L on large scale
video datasets (SSv2, Kinetics400), while having 3times less parameters,
12times smaller memory footprint, and 5times lower FLOPs count. Code and
checkpoints will be made available online at
https://github.com/google-deepmind/trecvit.
A new TRecViT architecture combines gated LRUs, self-attention, and MLPs for efficient video modeling, outperforming ViViT-L with reduced parameters, memory, and computational cost.
gated linear recurrent units LRUs self-attention MLPs causal model TRecViT ViViT-L SSv2 Kinetics400 parameter-efficient memory footprint FLOPs count