title: TRecViT: A Recurrent Video Transformer
summary: We propose a novel block for video modelling. It relies on a
  time-space-channel factorisation with dedicated blocks for each dimension:
  gated linear recurrent units (LRUs) perform information mixing over time,
  self-attention layers perform mixing over space, and MLPs over channels. The
  resulting architecture TRecViT performs well on sparse and dense tasks, trained
  in supervised or self-supervised regimes. Notably, our model is causal and
  outperforms or is on par with a pure attention model ViViT-L on large scale
  video datasets (SSv2, Kinetics400), while having 3times less parameters,
  12times smaller memory footprint, and 5times lower FLOPs count. Code and
  checkpoints will be made available online at
  https://github.com/google-deepmind/trecvit.
paper_authors._id: 6769b72361c7635a1e1e5dc0
paper_authors.name: Viorica Pătrăucean
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-01-06T10:47:22.333Z
paper_authors._id: 640b53d991d9f65c58a85d4a
paper_authors.avatarUrl: /avatars/bea43ffca19cb0705caea3b22f652adc.svg
paper_authors.fullname: Vio Patr
paper_authors.type: user
paper_authors.user: viorik
paper_authors._id: 6769b72361c7635a1e1e5dc1
paper_authors.name: Xu Owen He
paper_authors._id: 6769b72361c7635a1e1e5dc2
paper_authors.name: Joseph Heyward
paper_authors._id: 6769b72361c7635a1e1e5dc3
paper_authors.name: Chuhan Zhang
paper_authors._id: 6769b72361c7635a1e1e5dc4
paper_authors.name: Mehdi S. M. Sajjadi
paper_authors._id: 6769b72361c7635a1e1e5dc5
paper_authors.name: George-Cristian Muraru
paper_authors._id: 6769b72361c7635a1e1e5dc6
paper_authors.name: Artem Zholus
paper_authors._id: 6769b72361c7635a1e1e5dc7
paper_authors.name: Mahdi Karami
paper_authors._id: 6769b72361c7635a1e1e5dc8
paper_authors.name: Ross Goroshin
paper_authors._id: 6769b72361c7635a1e1e5dc9
paper_authors.name: Yutian Chen
paper_authors._id: 6769b72361c7635a1e1e5dca
paper_authors.name: Simon Osindero
paper_authors._id: 6769b72361c7635a1e1e5dcb
paper_authors.name: João Carreira
paper_authors._id: 6769b72361c7635a1e1e5dcc
paper_authors.name: Razvan Pascanu
paper_title: TRecViT: A Recurrent Video Transformer
paper_summary: We propose a novel block for video modelling. It relies on a
  time-space-channel factorisation with dedicated blocks for each dimension:
  gated linear recurrent units (LRUs) perform information mixing over time,
  self-attention layers perform mixing over space, and MLPs over channels. The
  resulting architecture TRecViT performs well on sparse and dense tasks, trained
  in supervised or self-supervised regimes. Notably, our model is causal and
  outperforms or is on par with a pure attention model ViViT-L on large scale
  video datasets (SSv2, Kinetics400), while having 3times less parameters,
  12times smaller memory footprint, and 5times lower FLOPs count. Code and
  checkpoints will be made available online at
  https://github.com/google-deepmind/trecvit.
paper_ai_keywords: gated linear recurrent units (LRUs) self-attention layers MLPs TRecViT ViViT-L SSv2 Kinetics400 FLOPs