summaries: This submission is about multi-modal representation learning from unlabeled data (self-sup.) using Transformer. This work deals with video, audio, and text data as multi-modal data. The approach to tackle representation learning is based on Transformer + contrastive learning. The contrastive learning in this work is interesting in that 1) noisy association is considered in multiple instances learning way [64], and 2) the different level of semantic granularity is carefully considered. Also, the authors adopt a simple DropToken to improve efficiency.  The training requires very large-scale training using HowTo100M and AudioSet. The authors demonstrated its effectiveness on action recognition, audio event classification, zero-shot video retrieval, image classification.  Multi-modal data is encoded by independent feature extractors, which is a small difference from other Vision Transformers. Naturally, the authors extend the positional encoding to 3D positional encoding (horizontal, vertical, and temporal axes) with relativeness.  Overall, the technical contribution is a bit weak, but this submission contains interesting analyses that are worthwhile to report in the community and also achieve state-of-the-art in downstream tasks.

summaries: This paper describes a self-supervised pure transformer-based multimodal representation learning including video-audio-text, i.e., VATT. VATT consists of  linear projection of input for each modality, modality-specific or agnostic transformer encoder, and multimodal projection head. VATT uses NCE loss for video-audio feature learning and MIL-NCE loss for text-video features. Also, the authors proposed DropToken as an efficient method for training.  They employ HowTo1M and AudioSet datasets for pretraining and evaluate their VATT for diverse finetuning tasks including video action recognition, audio event classification, and zero-shot text-to-video retrieval. With intensive experiments and ablation studies, they provide promising results and constructive analysis.

summaries: The paper proposes a transformer-based architecture for learning representations from video-audio-text triplet data without manual data annotation. The paper studies modality-specific transformers (one transformer for one modality) and modality-agnostic transformers (a shared transformer for all modalities). The learned representation is evaluated on various downstream tasks on image, video, audio and video-text.

summaries: This paper concerns multimodal self-supervised learning. It adopts the now ubiquitous contrastive learning framework to learn video, audio, text representation from each raw signal in joint embedding spaces. The proposed architecture heavily relies on MMV [1], with a twist on Visual/Audio Transformer encoders and modality-agnostic Transformers. Comprehensive experimental results are conducted on four diverse video(-text) tasks across ten datasets.

summaries: This paper provides a framework for self-supervised multimodal representations from unlabeled data with Transformer backbones. The authors study two styles of Transformer encoders: modality-specific or modality-agnostic.  Finally, they achieve good performance on various downstream tasks including action recognition, audio event classification, text-to-video retrieval and image classification.

abstractText: We present a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, our VideoAudio-Text Transformer (VATT) takes raw signals as inputs and extracts multimodal representations that are rich enough to benefit a variety of downstream tasks. We train VATT end-to-end from scratch using multimodal contrastive losses and evaluate its performance by the downstream tasks of video action recognition, audio event classification, image classification, and text-to-video retrieval. Furthermore, we study a modality-agnostic, single-backbone Transformer by sharing weights among the three modalities. We show that the convolution-free VATT outperforms state-of-the-art ConvNet-based architectures in the downstream tasks. Especially, VATT’s vision Transformer achieves the top-1 accuracy of 82.1% on Kinetics-400, 83.6% on Kinetics-600, 72.7% on Kinetics-700, and 41.1% on Moments in Time, new records while avoiding supervised pre-training. Transferring to image classification leads to 78.7% top-1 accuracy on ImageNet compared to 64.7% by training the same Transformer from scratch, showing the generalizability of our model despite the domain gap between videos and images. VATT’s audio Transformer also sets a new record on waveform-based audio event recognition by achieving the mAP of 39.4% on AudioSet without any supervised pre-training. VATT’s source code is publicly available.2

authors:
Jean-Baptiste Alayrac
Adrià Recasens
Rosalia Schneider
Relja Arandjelović
Jason Ramapuram
Jeffrey De Fauw
Lucas Smaira
Sander Dieleman
Andrew Zisserman

title: Selfsupervised multimodal versatile networks

venue: NeurIPS,

authors:
Humam Alwassel
Dhruv Mahajan
Bruno Korbar
Lorenzo Torresani
Bernard Ghanem
Du Tran

title: Self-supervised learning by cross-modal audio-video clustering

venue: arXiv preprint arXiv:1911.12667,

authors:
Relja Arandjelovic
Andrew Zisserman

title: Look, listen and learn

venue: In CVPR, 2017

title: Objects that sound

venue: In ECCV,

authors:
Anurag Arnab
Mostafa Dehghani
Georg Heigold
Chen Sun
Mario Lučić
Cordelia Schmid

title: Vivit: A video vision transformer

venue: arXiv preprint arXiv:2103.15691,

authors:
Dzmitry Bahdanau
Kyunghyun Cho
Yoshua Bengio

title: Neural machine translation by jointly learning to align and translate

venue: In ICLR,

authors:
Sagie Benaim
Ariel Ephrat
Oran Lang
Inbar Mosseri
William T Freeman
Michael Rubinstein
Michal Irani
Tali Dekel

title: Speednet: Learning the speediness in videos

venue: In CVPR,

authors:
Gedas Bertasius
Heng Wang
Lorenzo Torresani

title: Is space-time attention all you need for video understanding

venue: arXiv preprint arXiv:2102.05095,

authors:
Tom B Brown
Benjamin Mann
Nick Ryder
Melanie Subbiah
Jared Kaplan
Prafulla Dhariwal
Arvind Neelakantan
Pranav Shyam
Girish Sastry
Amanda Askell

title: Language models are few-shot learners

venue: arXiv preprint arXiv:2005.14165,

authors:
Nicolas Carion
Francisco Massa
Gabriel Synnaeve
Nicolas Usunier
Alexander Kirillov
Sergey Zagoruyko

title: End-to-end object detection with transformers

authors:
Mathilde Caron
Ishan Misra
Julien Mairal
Priya Goyal
Piotr Bojanowski
Armand Joulin

title: Unsupervised learning of visual features by contrasting cluster assignments

venue: In NeurIPS,

authors:
Joao Carreira
Andrew Zisserman

title: Quo vadis, action recognition? a new model and the kinetics dataset

authors:
Joao Carreira
Eric Noland
Andras Banki-Horvath
Chloe Hillier
Andrew Zisserman

title: A short note about kinetics-600

venue: arXiv preprint arXiv:1808.01340,

authors:
Mark Chen
Alec Radford
Rewon Child
Jeffrey Wu
Heewoo Jun
David Luan
Ilya Sutskever

title: Generative pretraining from pixels

venue: In ICML,

authors:
Ting Chen
Simon Kornblith
Mohammad Norouzi
Geoffrey Hinton

title: A simple framework for contrastive learning of visual representations

authors:
Xinlei Chen
Saining Xie
Kaiming He

title: An empirical study of training self-supervised visual transformers

venue: arXiv preprint arXiv:2104.02057,

authors:
Yen-Chun Chen
Linjie Li
Licheng Yu
Ahmed El Kholy
Faisal Ahmed
Zhe Gan
Yu Cheng
Jingjing Liu

title: Uniter: Universal image-text representation learning

authors:
Yunpeng Chen
Haoqi Fan
Bing Xu
Zhicheng Yan
Yannis Kalantidis
Marcus Rohrbach
Shuicheng Yan
Jiashi Feng

title: Drop an octave: Reducing spatial redundancy in convolutional neural networks with octave convolution

venue: In ICCV, 2019

authors:
Wei Dai
Chia Dai
Shuhui Qu
Juncheng Li
Samarjit Das

title: Very deep convolutional neural networks for raw waveforms

venue: In ICASSP,

authors:
Jia Deng
Wei Dong
Richard Socher
Li-Jia Li
Kai Li
Li Fei-Fei

title: Imagenet: A large-scale hierarchical image database

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova

title: Bert: Pre-training of deep bidirectional transformers for language understanding

venue: In NAACL, 2019

authors:
Carl Doersch
Abhinav Gupta
Alexei A Efros

title: Unsupervised visual representation learning by context prediction

venue: In ICCV,

authors:
Alexey Dosovitskiy
Lucas Beyer
Alexander Kolesnikov
Dirk Weissenborn
Xiaohua Zhai
Thomas Unterthiner
Mostafa Dehghani
Matthias Minderer
Georg Heigold
Sylvain Gelly
Jakob Uszkoreit
Neil Houlsby

title: An image is worth 16x16 words: Transformers for image recognition at scale

authors:
Heng Wang Du Tran
Lorenzo Torresani
Jamie Ray
Yann LeCun
Manohar Paluri

title: A closer look at spatiotemporal convolutions for action recognition. 2018 ieee

authors:
Quanfu Fan
Chun-Fu (Ricarhd) Chen
Hilde Kuehne
Marco Pistoia
David Cox

title: More Is Less: Learning Efficient Video Representations by Temporal Aggregation Modules

venue: In NeurIPS. 2019

authors:
William Fedus
Barret Zoph
Noam Shazeer

title: Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity

venue: arXiv preprint arXiv:2101.03961,

title: X3d: Expanding architectures for efficient video recognition

authors:
Christoph Feichtenhofer
Haoqi Fan
Jitendra Malik
Kaiming He

title: Slowfast networks for video recognition

authors:
Basura Fernando
Hakan Bilen
Efstratios Gavves
Stephen Gould

title: Self-supervised video representation learning with odd-one-out networks

authors:
Jonas Gehring
Michael Auli
David Grangier
Denis Yarats
Yann N Dauphin

title: Convolutional sequence to sequence learning

authors:
Jort F. Gemmeke
Daniel P.W. Ellis
Dylan Freedman
Aren Jansen
Wade Lawrence
R. Channing Moore
Manoj Plakal
Marvin Ritter

title: Audio set: An ontology and human-labeled dataset for audio events

authors:
Jort F Gemmeke
Daniel PW Ellis
Dylan Freedman
Aren Jansen
Wade Lawrence
R Channing Moore
Manoj Plakal
Marvin Ritter

authors:
Spyros Gidaris
Praveer Singh
Nikos Komodakis

title: Unsupervised representation learning by predicting image rotations

authors:
Rohit Girdhar
Deva Ramanan

title: Attentional pooling for action recognition

authors:
Jean-Bastien Grill
Florian Strub
Florent Altché
Corentin Tallec
Pierre H Richemond
Elena Buchatskaya
Carl Doersch
Bernardo Avila Pires
Zhaohan Daniel Guo
Mohammad Gheshlaghi Azar

title: Bootstrap your own latent: A new approach to self-supervised learning

authors:
Tengda Han
Weidi Xie
Andrew Zisserman

title: Video representation learning by dense predictive coding

venue: In ICCV Workshops,

title: Memory-augmented dense predictive coding for video representation learning

venue: In ECCV, 2020

authors:
Kaiming He
Haoqi Fan
Yuxin Wu
Saining Xie
Ross Girshick

title: Momentum contrast for unsupervised visual representation learning

authors:
Olivier J Hénaff
Aravind Srinivas
Jeffrey De Fauw
Ali Razavi
Carl Doersch
SM Eslami
Aaron van den Oord

title: Data-efficient image recognition with contrastive predictive coding

venue: arXiv preprint arXiv:1905.09272,

authors:
Dan Hendrycks
Kevin Gimpel

title: Gaussian error linear units (gelus)

venue: arXiv preprint arXiv:1606.08415,

authors:
Sepp Hochreiter
Jürgen Schmidhuber

title: Long short-term memory

venue: Neural computation,

authors:
Han Hu
Zheng Zhang
Zhenda Xie
Stephen Lin

title: Local relation networks for image recognition

authors:
Dahun Kim
Donghyeon Cho
In So Kweon

title: Self-supervised video representation learning with space-time cubic puzzles

venue: In AAAI,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

venue: arXiv preprint arXiv:1412.6980,

authors:
Dan Kondratyuk
Liangzhe Yuan
Yandong Li
Li Zhang
Mingxing Tan
Matthew Brown
Boqing Gong

title: Movinets: Mobile video networks for efficient video recognition

authors:
Qiuqiang Kong
Changsong Yu
Yong Xu
Turab Iqbal
Wenwu Wang
Mark D Plumbley

title: Weakly labelled audioset tagging with attention neural networks. TASLP, 2019

authors:
Qiuqiang Kong
Yin Cao
Turab Iqbal
Yuxuan Wang
Wenwu Wang
Mark D Plumbley

title: Panns: Large-scale pretrained audio neural networks for audio pattern recognition

venue: TASLP, 2020

authors:
Bruno Korbar
Du Tran
Lorenzo Torresani

title: Cooperative learning of audio and video models from self-supervised synchronization

venue: NeurIPS, 2018

authors:
Alex Krizhevsky
Ilya Sutskever
Geoffrey E Hinton

title: Imagenet classification with deep convolutional neural networks

authors:
H. Kuehne
H. Jhuang
E. Garrote
T. Poggio
T. Serre

title: HMDB: a large video database for human motion recognition

authors:
Yann LeCun
Léon Bottou
Yoshua Bengio
Patrick Haffner

title: Gradient-based learning applied to document recognition

venue: Proceedings of the IEEE,

authors:
Hsin-Ying Lee
Jia-Bin Huang
Maneesh Singh
Ming-Hsuan Yang

title: Unsupervised representation learning by sorting sequences

authors:
Jongpil Lee
Jiyoung Park
Keunhyoung Luke Kim
Juhan Nam

title: Sample-level deep convolutional neural networks for music auto-tagging using raw waveforms

venue: arXiv preprint arXiv:1703.01789,

authors:
William Lotter
Gabriel Kreiman
David Cox

title: Deep predictive coding networks for video prediction and unsupervised learning

venue: arXiv preprint arXiv:1605.08104,

authors:
Huaishao Luo
Lei Ji
Botian Shi
Haoyang Huang
Nan Duan
Tianrui Li
Xilin Chen
Ming Zhou

title: Univilm: A unified video and language pre-training model for multimodal understanding and generation

venue: arXiv preprint arXiv:2002.06353,

authors:
Antoine Miech
Dimitri Zhukov
Jean-Baptiste Alayrac
Makarand Tapaswi
Ivan Laptev
Josef Sivic

title: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

authors:
Antoine Miech
Jean-Baptiste Alayrac
Lucas Smaira
Ivan Laptev
Josef Sivic
Andrew Zisserman

title: End-to-end learning of visual representations from uncurated instructional videos

authors:
Tomas Mikolov
Kai Chen
Greg Corrado
Jeffrey Dean

title: Efficient estimation of word representations in vector space

venue: arXiv preprint arXiv:1301.3781,

authors:
Mathew Monfort
Alex Andonian
Bolei Zhou
Kandan Ramakrishnan
Sarah Adel Bargal
Tom Yan
Lisa Brown
Quanfu Fan
Dan Gutfreund
Carl Vondrick

title: Moments in time dataset: one million videos for event understanding

venue: TPAMI, 2019

authors:
Pedro Morgado
Nuno Vasconcelos
Ishan Misra

title: Audio-visual instance discrimination with cross-modal agreement

venue: arXiv preprint arXiv:2004.12943,

authors:
Mehdi Noroozi
Paolo Favaro

title: Unsupervised learning of visual representations by solving jigsaw puzzles

authors:
Deepak Pathak
Philipp Krahenbuhl
Jeff Donahue
Trevor Darrell
Alexei A Efros

title: Context encoders: Feature learning by inpainting

authors:
Mandela Patrick
Yuki M Asano
Polina Kuznetsova
Ruth Fong
João F Henriques
Geoffrey Zweig
Andrea Vedaldi

title: Multi-modal self-supervision from generalized data transformations

venue: arXiv preprint arXiv:2003.04298,

title: ESC: Dataset for Environmental Sound Classification

venue: In ACM MM,

authors:
AJ Piergiovanni
Anelia Angelova
Michael S Ryoo

title: Evolving losses for unsupervised video representation learning

authors:
Rui Qian
Tianjian Meng
Boqing Gong
Ming-Hsuan Yang
Huisheng Wang
Serge Belongie
Yin Cui

title: Spatiotemporal contrastive video representation learning

authors:
Zhaofan Qiu
Ting Yao
Chong-Wah Ngo
Xinmei Tian
Tao Mei

title: Learning spatio-temporal representation with local and global diffusion

authors:
Alec Radford
Karthik Narasimhan
Tim Salimans
Ilya Sutskever

title: Improving language understanding by generative pre-training

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI blog,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

authors:
Prajit Ramachandran
Niki Parmar
Ashish Vaswani
Irwan Bello
Anselm Levskaya
Jonathon Shlens

title: Stand-alone self-attention in vision models

venue: arXiv preprint arXiv:1906.05909,

authors:
Adrià Recasens
Pauline Luc
Jean-Baptiste Alayrac
Luyu Wang
Florian Strub
Corentin Tallec
Mateusz Malinowski
Viorica Patraucean
Florent Altché
Michal Valko

title: Broaden your views for self-supervised video learning

venue: arXiv preprint arXiv:2103.16559,

title: Factorization machines

venue: In ICDM,

authors:
Carlos Riquelme
Joan Puigcerver
Basil Mustafa
Maxim Neumann
Rodolphe Jenatton
André Susano Pinto
Daniel Keysers
Neil Houlsby

title: Scaling vision with sparse mixture of experts

venue: arXiv preprint arXiv:2106.05974,

authors:
Michael S Ryoo
AJ Piergiovanni
Mingxing Tan
Anelia Angelova

title: Assemblenet: Searching for multi-stream neural connectivity in video architectures

venue: arXiv preprint arXiv:1905.13209,

authors:
Gilad Sharir
Asaf Noy
Lihi Zelnik-Manor

title: An image is worth 16x16 words, what is a video worth

venue: arXiv preprint arXiv:2103.13915,

authors:
Noam Shazeer
Azalia Mirhoseini
Krzysztof Maziarz
Andy Davis
Quoc Le
Geoffrey Hinton
Jeff Dean

title: Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

venue: arXiv preprint arXiv:1701.06538,

authors:
Laura Smith
Nikita Dhawan
Marvin Zhang
Pieter Abbeel
Sergey Levine

title: Avid: Learning multi-stage tasks via pixel-level translation of human videos

venue: arXiv preprint arXiv:1912.04443,

authors:
Khurram Soomro
Amir Roshan Zamir
Mubarak Shah

title: Ucf101: A dataset of 101 human actions classes from videos in the wild

venue: arXiv preprint arXiv:1212.0402,

authors:
Nitish Srivastava
Elman Mansimov
Ruslan Salakhudinov

title: Unsupervised learning of video representations using lstms

authors:
Jonathan Stroud
David Ross
Chen Sun
Jia Deng
Rahul Sukthankar

title: D3d: Distilled 3d networks for video action recognition

venue: In WACV,

authors:
Chen Sun
Fabien Baradel
Kevin Murphy
Cordelia Schmid

title: Learning video representations using contrastive bidirectional transformer

venue: arXiv preprint arXiv:1906.05743,

authors:
Yonglong Tian
Dilip Krishnan
Phillip Isola

title: Contrastive multiview coding

authors:
Hugo Touvron
Matthieu Cord
Matthijs Douze
Francisco Massa
Alexandre Sablayrolles
Hervé Jégou

title: Training data-efficient image transformers & distillation through attention

venue: arXiv preprint arXiv:2012.12877,

authors:
Du Tran
Heng Wang
Lorenzo Torresani
Matt Feiszli

title: Video classification with channelseparated convolutional networks

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

venue: arXiv preprint arXiv:1706.03762,

authors:
Fei Wang
Mengqing Jiang
Chen Qian
Shuo Yang
Cheng Li
Honggang Zhang
Xiaogang Wang
Xiaoou Tang

title: Residual attention network for image classification

authors:
Jiangliu Wang
Jianbo Jiao
Linchao Bao
Shengfeng He
Yunhui Liu
Wei Liu

title: Selfsupervised spatio-temporal representation learning for videos by predicting motion and appearance statistics

title: Self-supervised video representation learning by pace prediction

authors:
Xiaofang Wang
Xuehan Xiong
Maxim Neumann
AJ Piergiovanni
Michael S Ryoo
Anelia Angelova
Kris M Kitani
Wei Hua

title: Attentionnas: Spatiotemporal attention cell search for video classification

authors:
Xiaolong Wang
Ross Girshick
Abhinav Gupta
Kaiming He

title: Non-local neural networks

authors:
Sanghyun Woo
Jongchan Park
Joon-Young Lee
In So Kweon

title: Cbam: Convolutional block attention module

authors:
Zhirong Wu
Yuanjun Xiong
Stella X Yu
Dahua Lin

title: Unsupervised feature learning via non-parametric instance discrimination

authors:
Saining Xie
Chen Sun
Jonathan Huang
Zhuowen Tu
Kevin Murphy

title: Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification

authors:
Dejing Xu
Jun Xiao
Zhou Zhao
Jian Shao
Di Xie
Yueting Zhuang

title: Self-supervised spatiotemporal learning via video clip order prediction

title: Msr-vtt: A large video description dataset for bridging video and language

authors:
Fuzhi Yang
Huan Yang
Jianlong Fu
Hongtao Lu
Baining Guo

title: Learning texture transformer network for image super-resolution

authors:
Mang Ye
Xu Zhang
Pong C Yuen
Shih-Fu Chang

title: Unsupervised embedding learning via invariant and spreading instance feature

authors:
Hongyi Zhang
Moustapha Cisse
Yann N Dauphin
David Lopez-Paz

title: mixup: Beyond empirical risk minimization

authors:
Richard Zhang
Phillip Isola
Alexei A Efros

title: Colorful image colorization

title: Split-brain autoencoders: Unsupervised learning by cross-channel prediction

title: Lecun. Character-level convolutional networks for text classification

title: Residual non-local attention networks for image restoration

authors:
Hengshuang Zhao
Jiaya Jia
Vladlen Koltun

title: Exploring self-attention for image recognition

authors:
Sixiao Zheng
Jiachen Lu
Hengshuang Zhao
Xiatian Zhu
Zekun Luo
Yabiao Wang
Yanwei Fu
Jianfeng Feng
Tao Xiang
Philip HS Torr

title: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers

title: Towards automatic learning of procedures from web instructional videos

heading: 1 Introduction

text:
Convolutional neural networks (CNNs) [53, 51] have triumphed over various computer vision tasks. The inductive bias induced by convolutions, namely translation invariance and locality, are proven effective for the visual data. In the meantime, however, we witness in the natural language processing (NLP) community a paradigm shift from the models with strong inductive biases, such as recurrent neural networks [43, 7] and CNNs [104, 32], to more general architectures constructed upon selfattention. Particularly, Transformers [88] have become the de facto model architecture for NLP
∗Work done during an internship at Google. 2https://github.com/google-research/google-research/tree/master/vatt
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
tasks [23, 70, 71, 10]. Pre-training a Transformer on large text corpora followed by fine-tuning gives rise to state-of-the-art results for different downstream tasks.
In view of the success of the attention mechanism in NLP, there has been a rich line of works exploring its potential in computer vision. Early work studied hybrid models consisting of both convolutions and attention modules [89, 94, 36, 105]. Recent studies showed that convolution-free, specially designed all-attention models can match CNNs’ performance on image recognition tasks [106, 44, 73]. Most recently, [25] achieved impressive performance on several image recognition tasks, including ImageNet [22], using a pre-trained Transformer with minimal architecture changes. Their work delivered a compelling message that “large scale (supervised) training trumps inductive bias (for image classification).” This conclusion was further extended to video recognition tasks by [9, 5].
However, the large-scale supervised training of Transformers is essentially troubling for two main reasons. First, it rules out the much larger other part of “big visual data,” i.e, the vast amount of unlabeled, unstructured visual data. As a result, the supervised training strategy could produce biased systems that require even more labeled data to correct their biases. Second, this strategy fundamentally limits the application scope of Transformers in computer vision because it is costly and extremely time-consuming to collect enough labeled images or videos for training the millions of parameters, choosing hyper-parameters, and validating their expected generalization.
Hence, this work poses another pressing question about the Transformers that take raw signals as input. How to empower them with large-scale, unlabeled visual data? To answer this question, we draw insights from NLP. BERT [23] and GPT [70, 71, 10] use masked language modeling as their pre-training tasks. Natural languages are organic supervision for Transformers. They sequentially place words, phrases, and sentences into context, granting them semantics and syntax. For visual data, the most organic supervision is arguably the multimodal videos. They are abundantly available in the digital world, and their temporal, cross-modality regulation, and therefore supervision, requires no human annotation. The extreme scale of multimodal videos is potentially capable to teach Transformers necessary priors, as opposed to predefined inductive biases, to model the visual world.
To this end, we study self-supervised, multimodal pre-training of three Transformers [88], which take as input the raw RGB frames of internet videos, audio waveforms, and text transcripts of the speech audio, respectively. We call the video, audio, text Transformers VATT. Figure 1 illustrates the architecture. VATT borrows the exact architecture from BERT [23] and ViT [25] except the layer of tokenization and linear projection reserved for each modality separately. This design shares the same spirit as ViT that we make the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks. Furthermore, the self-supervised, multimodal learning strategy resonates the spirit of BERT and GPT that the pre-training requires minimal human curated labels.
We evaluate the pre-trained Transformers on a variety of downstream tasks: image classification, video action recognition, audio event classification, and zero-shot text-to-video retrieval. Fine-tuning
the vision-modality Transformer on ImageNet [22] obtains the top-1 accuracy of 78.7%, which is comparable to 79.9% achieved by ViT. This result is especially appealing considering the domain gap between videos and images, and that ViT is pre-trained using a large-scale, human-curated image dataset. Furthermore, we set new records on Kinetics-400 [14], Kinetics-600 [15], Moments in Time [61], and AudioSet [33] without supervised pre-training.
Our VATT results, along with others reported for NLP tasks [23, 10], image recognition [25], semantic segmentation [108], point cloud classification [107], and action recoginition [9], demonstrate that Transformer is a versatile general-purpose architecture for different types of data.
To move one step forward, we challenge the Transformers in VATT by a seemingly too strong constraint: sharing weights among the video, audio, and text modalities. The idea is to test whether there exists a single, general-purpose model for all the modalities — of course, they still have their own layers of tokenization and linear projection. Preliminary results are encouraging. This modality-agnostic Transformer is on par with three modality-specific ones of slightly smaller sizes.
Finally, another contribution of this work is DropToken, a simple and yet effective technique to reduce the training complexity with a minor reduction of the end Transformers’ performance. DropToken randomly drops a portion of the video and audio tokens from each input sequence during training, allowing for high-resolution inputs and leveraging their abundance. This is significant for Transformers because their computational complexity is quadratic with respect to the number of input tokens.

heading: 2 Related work

heading: 2.1 Transformers in Vision

text: Transformer was originally built for NLP tasks [88] and the design of multi-head attention shows its effectiveness on modeling long-term correlation of words. A few attempts have been made to use Transformer for vision tasks like image super-resolution [99], object detection [11] and multimodal video understanding [84, 19, 57]. However these methods still rely on the feature extracted by CNNs. Recently, [25] proposes a set of convolution-free vision Transformers which directly work on raw images and obtain competitive performance with CNNs. [86] improves the training data efficiency of [25] by using stronger data augmentations and knowledge distillation. Since then, the pure Transformer design has been adopted to various vision tasks including semantic segmentation [108], point cloud classification [107], action recoginition [9, 78, 5]. To the best of our knowledge, our VATT is the first Transformer model on raw multimodal inputs of video, audio and text.

heading: 2.2 Self-Supervised Learning

text:
Single vision modality. Early work of self-supervised visual representation learning usually learns from unlabeled images via manually specified pretext tasks, like auto-encoding [64, 102, 103], patch location prediction [24], solving jigsaw puzzles [63], and image rotation prediction [35]. [95] propose a novel instance discrimination objective. The recent trend of contrastive learning [40, 17, 100, 37, 41, 85] integrates data augmentations and instance discrimination by maintaining relative consistency between representations of an image and its augmented view. Clustering can also provide an effective addition [12]. Recently, [18] conduct contrastive learning using ViT [25] and achieve impressive results. As for the video domain, it is natural to exploit the temporal signals as the pretext task. Examples include predicting the future frame [82], motion and appearance statistics [90], speed [8, 91] and encodings [56, 38, 39], sorting frames or video clips [54, 97, 45, 31]. Recently, [68] apply contrastive learning to videos with a temporal sampling strategy and temporally consistent spatial augmentation.
Multimodal video. Video is a natural source of multimodal data. Multimodal self-supervised learning can be achieved by predicting whether a video has correspondence with an audio stream [3, 4, 62, 50], cross-modality clustering [2], and evolving losses [67]. Recently, [1] use contrastive loss to learn from video, audio and text; [74] learn to predict a broad view that spans a longer temporal context from a narrow view. VATT serves as a first work combining the strength of convolution-free Transformer and multimodal contrastive learning.

heading: 3 Approach

text:
In this section, we introduce our convolution-free VATT architecture and elaborate on the selfsupervised multimodal objectives for training VATT from scratch.
Figure 1 is an overview of the architecture. We feed each modality to a tokenization layer, where the raw input is projected to an embedding vector followed by a Transformer. There are two major settings: 1) The backbone Transformers are separate and have specific weights for each modality, and 2) The Transformers share weights, namely, there is a single backbone Transformer applied to any of the modalities. In either setting, the backbone extracts modality-specific representations, which are then mapped to common spaces to be compared with each other by contrastive losses. We describe each module in the following.

heading: 3.1 Tokenization and Positional Encoding

text:
VATT operates on raw signals. The vision-modality input consists of 3-channel RGB pixels of video frames, the audio input is in the form of air density amplitudes (waveforms), and the text input is a sequence of words. We first define a modality-specific tokenization layer that takes as input the raw signals and returns a sequence of vectors to be fed to the Transformers. Besides, each modality has its own positional encoding, which injects the order of tokens into Transformers [88]. We partition an entire video clip of size T ×H ×W to a sequence of ⌈T/t⌉ · ⌈H/h⌉ · ⌈W/w⌉ patches, where each patch contains t × h × w × 3 voxels. We apply a linear projection on the entire voxels in each patch to get a d-dimensional vector representation. This projection is performed by a learnable weight Wvp ∈ Rt·h·w·3×d. This can be seen as a 3D extension of the patching mechanism proposed in [25]. To encode the position of these patches, we define a dimension-specific sequence of learnable embeddings as follows:
ei,j,k = eTemporali+eHorizontalj + eVerticalk,
ETemporal ∈ R⌈T/t⌉×d, EHorizontal ∈ R⌈H/h⌉×d, EVertical ∈ R⌈W/w⌉×d (1)
where ei is the i-th row of E. This scheme allows us to use ⌈T/t⌉+ ⌈H/h⌉+ ⌈W/w⌉ positional embeddings to encode all the ⌈T/t⌉ · ⌈H/h⌉ · ⌈W/w⌉ patches in a video clip. The raw audio waveform is a 1D input with length T ′, and we partition it to ⌈T ′/t′⌉ segments each containing t′ waveform amplitudes. Similar to video, we apply a linear projection with a learnable weight Wap ∈ Rt
′×d to all elements in a patch to get a d-dimensional vector representation. We use ⌈T ′/t′⌉ learnable embeddings to encode the position of each waveform segment. For text, we first construct a vocabulary of size v out of all words in our training dataset. For an input text sequence, we then map each word to a v-dimensional one-hot vector followed by a linear projection with a learnable weight Wtp ∈ Rv×d. This is equivalent to an embedding dictionary lookup, which has been widely used in natural language understanding [60].

heading: 3.1.1 DropToken

text: We introduce DropToken, a simple and yet effective strategy to reduce the computational complexity during training. Once we get the token sequence for the video or audio modality, we randomly sample a portion of the tokens and then feed the sampled sequence, not the complete set of tokens, to the Transformer. This is crucial for reducing the computational cost because a Transformer’s computation complexity is quadratic, O(N2), where N is number of tokens in the input sequence. Any effort on reducing the input length would reduce the number of FLOPs quadratically. This has an immediate impact on the wall clock time for training these models and makes it possible to host large models in limited hardware. We argue that instead of reducing the resolution or dimension of the raw inputs, it is better to keep a high-fidelity input and randomly sample the tokens via DropToken. DropToken is appealing especially with the raw video and audio inputs, which may contain high redundancies.

heading: 3.2 The Transformer Architecture

text:
For simplicity, we adopt the most established Transformer architecture [23], which has been widely used in NLP. Similar to ViT [25], we do not tweak the architecture so that our weights can be easily transferred to any standard Transformer implementation. We will briefly elaborate on the pipeline (also illustrated in Figure 1 middle panel) and refer the reader to [25, 23] for more details of the
standard Transformer architecture. The sequence of input tokens to the Transformer follows the below formulation:
zin = [xAGG; x0WP ; x1WP ; . . . ; xNWP ] + ePOS (2)
where xn is the input patches sequence and xAGG is the learnable embedding of a special aggregation token whose corresponding output in the Transformer (z0out) is used as the aggregated representation for the entire input sequence. This will be later used for classification and common space mapping. We use a standard self-attention [88] as the Multi-Head-Attention (MHA) module, and GeLU [42] as the activation in the MLP layer. We also use Layer Normalization [6] before the MHA and MLP modules. In our text model, we remove the position encoding ePOS and add a learnable relative bias to each attention score of the first layer in the MHA module. This simple change makes our text model’s weights directly transferable to the state-of-the-art text model T5 [72].

heading: 3.3 Common Space Projection

text:
We use common space projection and contrastive learning in that common space to train our networks. More specifically, given a video-audio-text triplet, we define a semantically hierarchical common space mapping that enables us to directly compare video-audio pairs as well as video-text pairs by the cosine similarity. As argued in [1], such comparison is more feasible if we assume there are different levels of semantic granularity for these modalities. To achieve this, we define multi-level projections as follows:
zv,va = gv→va(z video out ), za,va = ga→va(z audio out )
zt,vt = gt→vt(z text out ), zv,vt = gv→vt(zv,va)
(3)
where gv→va and ga→va are the projection heads to respectively map the video and audio Transformers’ outputs to the video-audio common space Sva. Moreover, gt→vt and gv→vt project the text Transformer’s outputs and the video embedding in the Sva space to video-text common space, Svt. This multi-level common space projection is depicted in Figure 1 (the rightmost panel). The main intuition behind this hierarchy is that different modalities have different levels of semantic granularity, so we should impose this as an inductive bias in the common space projection. Similar to [1], we use a linear projection for ga→va(.), gt→vt(.), and gv→vt(.), and a two-layer projection with ReLU in between for gv→va(.). To ease the training, a batch normalization is used after each linear layer.

heading: 3.4 Multimodal Contrastive Learning

text:
Inspired by [1, 3, 59], we use Noise Contrastive Estimation (NCE) to align video-audio pairs and Multiple Instance Learning NCE (MIL-NCE) to align video-text pairs. The pairs are composed from different temporal locations in the video-audio-text stream. Positive pairs from two modalities are constructed by sampling their corresponding streams from the same location in the video, and negative pairs are constructed by sampling from any non-matching locations in the video [1]. Concretely, given the common space specified in Section 3, the loss objectives can be written as follows:
NCE(zv,va, za,va) = − log
( exp(z⊤v,vaza,va/τ)
exp(z⊤v,vaza,va/τ) + ∑ z′∈N exp(z ′⊤ v,vaz ′ a,va/τ)
) , (4)
MIL-NCE(zv,vt, {zt,vt}) = − log
( ∑ zt,vt∈P exp(z
⊤ v,vtzt,vt/τ)∑
zt,vt∈P exp(z ⊤ v,vtzt,vt/τ) + ∑ z′∈N exp(z ′⊤ v,vtz ′ t,vt/τ)
) ,
(5) where N contains all non-matching pairs in a batch. In Equation 5, P contains five text clips that are nearest neighbors to the video clip in time. τ is a temperature to adjust the softness of the objectives in distinguishing the positive pairs from the negative pairs.
The overall per-sample objective for training the entire VATT model end-to-end is as follows:
L = NCE(zv,va, za,va) + λMIL-NCE(zv,vt, {zt,vt}), (6)
where λ balances the two losses. The model is optimized based on the back-propagation of the average loss calculated over a batch of samples.

heading: 4 Experiments

text: In this section, we first briefly describe the experimental setup for the pre-training and downstream evaluation, and then present the results and analytic interpretation of VATT in different tasks. We refer the reader to the Appendix for a more detailed description of all experimental settings.

heading: 4.1 Experimental Setup

text:
Pre-train: we use a combination of AudioSet [33] and HowTo100M [58] datasets to pre-train VATT— we use only a subset of the HowTo100M dataset in compliance with Youtube’s policies. Following [1], we use video-audio-text triplets from HowTo100M clips while only using video-audio pairs from AudioSet. We sample 32 frames at 10 fps with a spatial size of 224 × 224 following a random crop, horizontal flip and color augmentation (details in A.2.1). Accordingly, we sample audio waveforms in sync at 48kHz. Both video and audio are normalized between [-1,1]. We use patch sizes of 4× 16× 16 and 128 for video and raw waveform tokenization, respectively (ablation in A.5). We use one-hot vectors to encode text sequences (capped to 16 tokens) with the vocabulary size of 216. In all pre-training experiments, we use DropToken with drop rate 50%. We train our models using the Adam optimizer [46] with a quarter-period cosine scheduled learning rate from 1e-4 to 5e-5 and 10k warmup steps. Optimization is performed on totally 500k steps with batch size 2048 (512 in exploration experiments). Following the previously established practice [1] for the projection to the common spaces Sva and Svt, we use dva = 512 and dvt = 256. We also use the temperature of τ = 0.07 and the weight of λ = 1 in the loss in Equation 6. We use 4 network sizes in our experiments (details in A.2.2). We use the Medium model (155M parameters) for our modality-agnostic variant (VATT-MA), and 3 variants for the modality-specific video-audio-text backbones: Base-Base-Small (BBS; 197M), Medium-Base-Small (MBS; 264M), and Large-Base-Small (LBS; 415M). Pre-training an MBS VATT with batch size 2048 on 256 TPUs (v3) takes less than 3 days. Pre-training with batch size 512 takes less than 1 day.
Downstream: we evaluate the pre-trained VATT models on 4 major downstream tasks using a total of 10 datasets. We use UCF101 [81], HMDB51 [52], Kinetics-400 [14], Kinetics-600 [15], and Moments in Time [61] for video action recognition. We use ESC50 [66] and AudioSet [33] for audio event classification, and we evaluate the quality of our video-text common space representations by zero-shot text-to-video retrieval on YouCook2 [109] and MSR-VTT [98]. Finally, we evaluate the transferability of the vision backbone by fine-tuning it on ImageNet classification [22]. Since HMDB51, UCF101, and ESC50 are very small datasets compared to the size of our networks, we only use them to train a linear classifier on top of the frozen pre-trained backbones. In our exploration experiments, we report linear classification accuracy and zero-shot video retrieval metrics. We refer to the Appendix for a detailed description of the datasets and the experimental setup.

heading: 4.2 Results

heading: 4.2.1 Fine-tuning for video action recognition

text:
We fine-tune VATT’s vision Transformer on Kinetics-400, Kinetics-600, and Moments in Time, three of the arguably most established large-scale datasets for video action recognition. We use the final checkpoints of four pre-train settings for these experiments: three modality-specific variations (LBS, MBS, BBS), and one modality-agnostic (Medium). Table 1 shows the results compared with the state-of-the-art video models. On all three datasets, we achieve higher accuracy than previous works including TimeSFormer [9], a recent effort in fine-tuning the ViT checkpoints obtained by supervised pre-training. In contrast, our pre-training does not rely on any labels curated by humans. To the best of our knowledge, VATT provides the first vision Transformer backbone that is pre-trained from scratch using self-supervision on multimodal videos and achieves state-of-the-art results on video action recognition. It is also worth mentioning that fine-tuning VATT on the most recent Kinetics-700 dataset results in a top-1 accuracy of 72.7%, which outperforms the state-of-the-art top-1 accuracy of 72.4% in [47].
To further quantify how much the multimodal self-supervised pre-training helps in achieving these numbers, we train a variant from scratch without any pre-training and observe the top-1 and top-5 accuracies of 26.4% and 51.8% on Kinetics-400, respectively. The low accuracies verify the efficacy of our pre-training strategy for VATT. Finally, we find that VATT-MA-Medium, the modality-agnostic
backbone shared by the video, audio, and text modalities, is on par with the modality-specific VATTBase when fine-tuned for the video action recognition. This result is encouraging as it indicates the potential of unifying three data modalities by a single Transformer backbone.

heading: 4.2.2 Fine-tuning for audio event classification

text: We fine-tune VATT’s audio Transformer on AudioSet, which benchmarks the task of multi-label audio event classification. We use the final checkpoints of two pre-train settings: one modality-specific (BBS), and one modality-agnostic (Medium). Table 2 shows the results compared to state-of-the-art models. Following common practice [34, 48], we report mean Average Precision (mAP), Area Under Curve (AUC), and d-prime (based on AUC) [34]. Our audio Transformer consistently outperforms the existing CNN-based models in all metrics. More interestingly, fine-tuning the modality-agnostic backbone (VATT-MA-Medium) is on par with fine-tuning the modality-specific one (VATT-Base). To the best of our knowledge, VATT is the first Transformer that outperforms CNN-based models in audio event recognition. VATT operates on raw waveforms and does not utilize any handcrafted features.

heading: 4.2.3 Fine-tuning for image classification

text: In this section, we show that our pipeline is capable of transferring the learned knowledge into another domain by performing the image classification task, even though the models are pre-trained in the multimodal video domain. We fine-tune the vision Transformer in VATT-BBS on ImageNet without any modification to the backbone architecture. Instead, to satisfy the voxel-to-patch layer we replicate the input image 4 times and feed it to the network. The network sees the input as a single-frame video clip and performs spatial self-attention. Table 3 shows the results for fine-tuning the vision Transformer end-to-end on ImageNet. We can see that our pre-training leads to a significant boost in the accuracy compared to training from scratch. We also observe that even though the self-supervised pre-training happens in the video domain, we still achieve competitive results to the supervised pre-training using large-scale image data [25].

heading: 4.2.4 Zero-shot text-to-video retrieval

text:
We feed video-text pairs to VATT-MBS, and extract representations in the Svt space. We then calculate the similarity between each video-text pair from YouCook2 and MSR-VTT. Given a text query, we rank the videos based on their similarities to the text. We then measure the recall for the
METHOD PRE-TRAINING DATA TOP-1 TOP-5
iGPT-L [16] ImageNet 72.6 - ViT-Base [25] JFT 79.9 -
VATT-Base - 64.7 83.9 VATT-Base HowTo100M 78.7 93.9
correct video in the top-10 videos. We also measure the median of the rank of the correct video. Table 4 compares our video retrieval results to two baselines. In our experiments we observe that the zero-shot retrieval results are heavily affected by the batch size and number of epochs, confirming the observation made in [1]. That said, our model still delivers comparable results to MMV [1] while being pre-trained with a half number of epochs and a half batch size of theirs. We also experiment with a larger batch size 8192 and longer pre-training for 6 epochs, arriving at exactly the same results as MIL-NCE [59] on YouCook2 and the R@10 of 29.2 and MedR of 42 on MSR-VTT. We also notice that, probably due to the noisy nature of text transcripts, a sophisticated language model like ours is underrated. As shown in [1], using a simple linear projection would still perform reasonably well. It is worth exploring other, higher-quality text sources in future work.

heading: 4.2.5 Feature visualization

text:
We take our modality-specific and modality-agnostic VATT fine-tuned on Kinetics-400 and visualize their output feature representations using t-SNE. For comparison, we also include the feature visualization of the vision Transformer trained from scratch on Kinetics-400. From Figure 2, we observe that the fine-tuned VATT yields a much better separation than the model trained from scratch. Furthermore, it is worth noting that there is no clear difference between the modality-agnostic features and the modality-specific ones.
We further investigate the VATT backbones without any fine-tuning. We randomly choose 1k video clips from the YouCook2 dataset and store the representations from two points of a pre-trained VATT model. One is after the tokenization layer (input space of the Transformer), and the other is after the common space projection (output space), where the loss is computed. Figure 3-top visualizes the representations, comparing modality-specific VATT to modality-agnostic VATT. Interestingly, we observe that the representations are slightly more mixed together in the modality-agnostic setting compared to the modality-specific ones, implying that the modality-agnostic backbone sees different modalities as different symbols describing the same concept. This is analogous to a unified language model in NLP that supports multiple languages.
To see how well VATT distinguishes positive video-text pairs from randomly sampled pairs, we calculate pair-wise similarities for all possible pairs and perform a Kernel Density Estimation (KDE) to visualize the distributions of the similarities of the positive pairs vs. negative pairs. We perform this procedure for both input and output spaces of the modality-specific and modality-agnostic backbones. Figure 3-bottom shows the KDE curves of these similarities. We can see that VATT in both settings separates the positive and negative pairs in its output space. This verifies VATT’s efficacy in learning a semantic common space for different modalities, even if we share the backbone across modalities.

heading: 4.2.6 Model Activations

text:
We measure the average activation of the modality-agnostic VATT when a full multimodal input is fed to the model. More specifically, we sample 100k short video clips from the test split of HowTo100M along with their corresponding audio and text and feed them to the model separately. For each
From Scratch
Modality-Specific
t-S NE
-3
Modality-Agnostic
modality, we calculate the average activation of each node at the output of the MLP module, before the residual addition (Figure 1-Transformer Encoder). Figure 4 shows the average activations across all nodes in a Medium-size model. We observe that earlier nodes in the model are activated with the text inputs, while the middle-to-later nodes are activated with video and audio modalities. However, the nodes in the last layers of the network are activated with all modalities almost equally. This might suggest that the model allocates different nodes to certain modalities while reaching the same level of semantic perception for all modalities in the later layers. Such observation encourages further studies on the possibility of utilizing Mixture-of-Experts [79, 28, 76] to increase the model’s capacity for simultaneous multimodal perception. We leave this direction of research for future work.

heading: 4.2.7 Effect of DropToken

text: We introduced a new method to reduce the redundancy in high-resolution data. To study the effect of the proposed DropToken method on downstream applications and the pre-training computation, we perform pre-training by randomly dropping 75%, 50%, 25%, and 0% (no drop) of the tokens from the video and audio inputs. Table 5 shows the accuracy of linear classification on HMDB51, UCF101, ESC50 and R@10 on YouCook2 and MSR-VTT vs. the drop rate along with GFLOPs during a forward call. We choose 50% sampling rate for our large-scale pre-training as it offers a good trade-off between accuracy and computational costs. We then take the final checkpoint of the pre-trained VATT with 50% DropToken rate and perform fine-tuning on Kinetics-400 at different DropToken rates and at different spatial and temporal resolutions to see how high-resolution inputs coupled with DropToken compare to low-resolution inputs with no tokens dropped during fine-tuning. Table 6 shows the top-1 accuracy on Kinetics-400. We argue against using low-resolution inputs, which is the most common approach to reduce the computational cost during training. Instead, we suggest using high-resolution inputs with DropToken, whose accuracy and training cost are comparable to or better than low-resolution counterparts.

heading: 5 Conclusion and Discussion

text:
In this paper, we present a self-supervised multimodal representation learning framework based on Transformers. Our study suggests that Transformers are effective for learning semantic video/audio/text representations — even if one model is shared across modalities — and multimodal self-supervised pre-training is promising for reducing their dependency on large-scale labeled data. We show that DropToken can significantly reduce the pre-training complexity with video and audio modalities and have minor impact on the models’ generalization. We report new records of results on video action recognition and audio event classification and competitive performance on image classification and video retrieval. Having these results, we still see some limitations in our work. Firstly, not all videos have organic audio or speech, while our approach depends on meaningful multimodal correspondences. Besides, the text modality currently consists of speech transcripts, which are noisy and sometimes sparse. Potential negative Societal Impacts are mainly concerned with applications. The models could be biased if one applies our approach to the multimodal videos that are not representative enough. Finally, our method is still demanding in computation, though we managed to avoid the need for human labels. Future work can improve upon these limitations.
Acknowledgments and Disclosure of Funding
We would like to thank Min-Hsuan Tsai, Jean-Baptise Alayrac, Andrew Audibert, Yeqing Li, Vidush Mukund, and the TensorFlow team for their help with codes, infrastructure, and insightful discussions.

title: VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text