summaries: The paper is a nice read. It builds on a line of research on multi-modal video understanding that utilises transformers where these works: 1) fix one of the transformer models (e.g. BERT) and 2) utilise tokens and thus do not train the approach in an end-to-end fashion. This typical trend is due to the memory requirements for training a multi-modal transformer end-to-end. In this work, the authors present a method for learning audiovisual (AV) representations from videos using a Transformer-based model architecture. Since both video processing and Transformer-based model are memory-intensive, a parameter-reducing scheme is proposed, which facililates training the model end-to-end. The AV representations are learned by training the network to solve two self-supervised pertaining tasks, and subsequently evaluated on various audio/visual downstream tasks. An ablation analysis is performed to demonstrate the efficacy of the various contributions. This paper studies modeling and training choices when designing a single model based on ConvNets and transformers for audio-visual representation learning. It proposes ablations for which weights/layers to share across modalities, when/where to fuse/join both modalities, and other modeling details (that matter). It also completes pre-training with 3 InfoNCE (audio-audio, visual-visual, audio-visual) with a binary classification loss about if two pairs of audio-visual are from the same or different videos. As strategies for negative sampling in audio and videos are different, it proposes to sample negatives that are similar in the ConvNets' embeddings. The models are pretrained on Kinetics-700 and AudioSet and evaluated on UCF101, ESC-50, and Kinectics-Sounds.
abstractText: The recent success of Transformers in the language domain has motivated adapting it to a multimodal setting, where a new visual model is trained in tandem with an already pretrained language model. However, due to the excessive memory requirements from Transformers, existing work typically fixes the language model and train only the vision module, which limits its ability to learn cross-modal information in an end-to-end manner. In this work, we focus on reducing the parameters of multimodal Transformers in the context of audio-visual video representation learning. We alleviate the high memory requirement by sharing the parameters of Transformers across layers and modalities; we decompose the Transformer into modality-specific and modality-shared parts so that the model learns the dynamics of each modality both individually and together, and propose a novel parameter sharing scheme based on low-rank approximation. We show that our approach reduces parameters of the Transformers up to 97%, allowing us to train our model end-to-end from scratch. We also propose a negative sampling approach based on an instance similarity measured on the CNN embedding space that our model learns together with the Transformers. To demonstrate our approach, we pretrain our model on 30-second clips (480 frames) from Kinetics-700 and transfer it to audio-visual classification tasks.
authors.name: Sangho Lee
authors.name: Youngjae Yu
authors.name: Gunhee Kim
references.authors: Chris Alberti Jeffrey Ling Michael Collins David Reitter
references.title: Fusion of Detected Objects in Text for Visual Question Answering
references.venue: EMNLP-IJCNLP,
references.authors: Humam Alwassel Dhruv Mahajan Lorenzo Torresani Bernard Ghanem Du Tran
references.title: SelfSupervised Learning by Cross-Modal Audio-Video Clustering
references.venue: arXiv preprint arXiv:1911.12667,
references.authors: Relja Arandjelovic Andrew Zisserman
references.title: Objects that Sound
references.venue: In ECCV,
references.authors: Yusuf Aytar Carl Vondrick Antonio Torralba
references.title: SoundNet: Learning Sound Representations from Unlabeled Video
references.venue: In NeurIPS,
references.authors: Shaojie Bai J. Zico Kolter Vladlen Koltun
references.title: Deep Equilibrium Models
references.venue: In NeurIPS,
references.authors: Shaojie Bai J Zico Kolter Vladlen Koltun
references.title: Trellis Networks for Sequence Modeling
references.venue: In ICLR,
references.authors: Tadas Baltrušaitis Chaitanya Ahuja Louis-Philippe Morency
references.title: Multimodal Machine Learning: A Survey and Taxonomy
references.venue: IEEE Transactions on Pattern Analysis and Machine Intelligence,
references.authors: Wim Boes Hugo Van hamme
references.title: Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events
references.venue: In ACM MM,
references.authors: Mathilde Caron Ari Morcos Piotr Bojanowski Julien Mairal Armand Joulin
references.title: Pruning Convolutional Neural Networks with Self-Supervision
references.venue: arXiv preprint arXiv:2001.03554,
references.authors: Joao Carreira Eric Noland Chloe Hillier Andrew Zisserman
references.title: A Short Note on the Kinetics-700 Human Action Dataset
references.authors: Beidi Chen Weiyang Liu Animesh Garg Zhiding Yu Anshumali Shrivastava Jan Kautz Anima Anandkumar
references.title: Angular Visual Hardness
references.venue: In ICML Workshop on Identifying and Understanding Deep Learning Phenomena,
references.authors: Ting Chen Simon Kornblith Mohammad Norouzi Geoffrey Hinton
references.title: A Simple Framework for Contrastive Learning of Visual Representations
references.venue: arXiv preprint arXiv:2002.05709,
references.authors: Yen-Chun Chen Linjie Li Licheng Yu Ahmed El Kholy Faisal Ahmed Zhe Gan Yu Cheng Jingjing Liu
references.title: UNITER: UNiversal Image-TExt Representation Learning
references.venue: arXiv preprint arXiv:1909.11740,
references.authors: Kyunghyun Cho Bart Van Merriënboer Caglar Gulcehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio
references.title: Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
references.venue: In EMNLP,
references.authors: Raj Dabre Atsushi Fujita
references.title: Recurrent Stacking of Layers for Compact Neural Machine Translation Models
references.venue: In AAAI,
references.authors: Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Łukasz Kaiser
references.title: Universal Transformers
references.venue: In ICLR,
references.authors: Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova
references.title: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
references.venue: In NAACL-HLT,
references.authors: Ariel Ephrat Inbar Mosseri Oran Lang Tali Dekel Kevin Wilson Avinatan Hassidim William T Freeman Michael Rubinstein
references.title: Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
references.venue: ACM Transactions on Graphics,
references.authors: Christoph Feichtenhofer Haoqi Fan Jitendra Malik Kaiming He
references.title: SlowFast Networks for Video Recognition
references.venue: In ICCV,
references.authors: Ruohan Gao Kristen Grauman
references.title: 2.5D Visual Sound
references.venue: In CVPR,
references.authors: Jort F. Gemmeke Daniel P.W. Ellis Dylan Freedman Aren Jansen Wade Lawrence R. Channing Moore Manoj Plakal Marvin Ritter
references.title: Audio Set: An Ontology and Human-Labeled Dataset for Audio Events
references.authors: Chunhui Gu Chen Sun David A Ross Carl Vondrick Caroline Pantofaru Yeqing Li Sudheendra Vijayanarasimhan George Toderici Susanna Ricco Rahul Sukthankar
references.title: AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions
references.authors: Michael Gutmann Aapo Hyvärinen
references.title: Noise-Contrastive Estimation: A New Estimation Principle for Unnormalized Statistical Models
references.venue: In AISTATS,
references.authors: Tengda Han Weidi Xie Andrew Zisserman
references.title: Video Representation Learning by Dense Predictive Coding
references.venue: In ICCV Workshop on Large Scale Holistic Video Understanding,
references.authors: Kaiming He Xiangyu Zhang Shaoqing Ren Jian Sun
references.title: Deep Residual Learning for Image Recognition
references.authors: Dan Hendrycks Kevin Gimpel
references.title: Gaussian Error Linear Units (GELUs)
references.venue: arXiv preprint arXiv:1606.08415,
references.authors: R Devon Hjelm Alex Fedorov Samuel Lavoie-Marchildon Karan Grewal Phil Bachman Adam Trischler Yoshua Bengio
references.title: Learning deep representations by mutual information estimation and maximization
references.venue: In ICLR,
references.authors: Sepp Hochreiter Jürgen Schmidhuber
references.title: Long Short-Term Memory
references.venue: Neural computation,
references.authors: Andrew Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen Mingxing Tan Weijun Wang Yukun Zhu Ruoming Pang Vijay Vasudevan
references.title: Searching for MobileNetV3
references.authors: Andrew G Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand Marco Andreetto Hartwig Adam
references.title: MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
references.venue: arXiv preprint arXiv:1704.04861,
references.authors: Jeremy Howard Sebastian Ruder
references.title: Universal Language Model Fine-tuning for Text Classification
references.venue: In ACL,
references.authors: Di Hu Feiping Nie Xuelong Li
references.title: Deep Multimodal Clustering for Unsupervised Audiovisual Learning
references.authors: Forrest N Iandola Song Han Matthew W Moskewicz Khalid Ashraf William J Dally Kurt Keutzer
references.title: SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
references.venue: arXiv preprint arXiv:1602.07360,
references.authors: Longlong Jing Xiaodong Yang Jingen Liu Yingli Tian
references.title: Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
references.venue: arXiv preprint arXiv:1811.11387,
references.authors: Will Kay Joao Carreira Karen Simonyan Brian Zhang Chloe Hillier Sudheendra Vijayanarasimhan Fabio Viola Tim Green Trevor Back Paul Natsev
references.title: The Kinetics Human Action Video Dataset
references.venue: arXiv preprint arXiv:1705.06950,
references.authors: Evangelos Kazakos Arsha Nagrani Andrew Zisserman Dima Damen
references.title: EPIC-Fusion: AudioVisual Temporal Binding for Egocentric Action Recognition
references.venue: In ICCV,
references.authors: Dahun Kim Donghyeon Cho In So Kweon
references.title: Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles
references.venue: In AAAI,
references.authors: Bruno Korbar Du Tran Lorenzo Torresani
references.title: Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
references.venue: In NeurIPS,
references.authors: Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut
references.title: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
references.venue: In ICLR,
references.authors: Mario Lezcano-Casado David Martínez-Rubio
references.title: Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Group
references.authors: Gen Li Nan Duan Yuejian Fang Min Gong Daxin Jiang Ming Zhou
references.title: Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
references.venue: In AAAI,
references.authors: Liunian Harold Li Mark Yatskar Da Yin Cho-Jui Hsieh Kai-Wei Chang
references.title: VisualBERT: A Simple and Performant Baseline for Vision and Language
references.authors: Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer Veselin Stoyanov
references.title: RoBERTa: A Robustly Optimized BERT Pretraining Approach
references.authors: Ilya Loshchilov Frank Hutter
references.title: Decoupled Weight Decay Regularization
references.venue: In ICLR,
references.authors: Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee
references.title: ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
references.venue: In NeurIPS,
references.authors: Huaishao Luo Lei Ji Botian Shi Haoyang Huang Nan Duan Tianrui Li Xilin Chen Ming Zhou
references.title: UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation
references.venue: arXiv preprint arXiv:2002.06353,
references.authors: Antoine Miech Jean-Baptiste Alayrac Lucas Smaira Ivan Laptev Josef Sivic Andrew Zisserman
references.title: End-to-End Learning of Visual Representations from Uncurated Instructional Videos
references.authors: Pedro Morgado Nuno Vasconcelos Ishan Misra
references.title: Audio-Visual Instance Discrimination with Cross-Modal Agreement
references.venue: arXiv preprint arXiv:2004.12943,
references.authors: Jiquan Ngiam Aditya Khosla Mingyu Kim Juhan Nam Honglak Lee Andrew Y Ng
references.title: Multimodal Deep Learning
references.venue: In ICML,
references.authors: Aaron van den Oord Yazhe Li Oriol Vinyals
references.title: Representation Learning with Contrastive Predictive Coding
references.venue: arXiv preprint arXiv:1807.03748,
references.authors: Andrew Owens Alexei A Efros
references.title: Audio-Visual Scene Analysis with Self-Supervised Multisensory Features
references.venue: In ECCV,
references.authors: Andrew Owens Phillip Isola Josh McDermott Antonio Torralba Edward H Adelson William T Freeman
references.title: Visually Indicated Sounds
references.authors: Andrew Owens Jiajun Wu Josh H McDermott William T Freeman Antonio Torralba
references.title: Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
references.venue: International Journal of Computer Vision,
references.authors: Daniel S Park William Chan Yu Zhang Chung-Cheng Chiu Barret Zoph Ekin D Cubuk Quoc V Le
references.title: SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
references.authors: Mandela Patrick Yuki M Asano Ruth Fong João F Henriques Geoffrey Zweig Andrea Vedaldi
references.title: Multi-modal Self-Supervision from Generalized Data Transformations
references.venue: arXiv preprint arXiv:2003.04298,
references.authors: Matthew E Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer
references.title: Deep Contextualized Word Representations
references.authors: Karol J. Piczak
references.title: ESC: Dataset for Environmental Sound Classification
references.venue: In ACM-MM,
references.authors: Karol J. Piczak
references.title: Environmental Sound Classification with Convolutional Neural Networks
references.venue: In MLSP,
references.authors: AJ Piergiovanni Anelia Angelova Michael S Ryoo
references.title: Evolving Losses for Unsupervised Video Representation Learning
references.venue: In CVPR,
references.authors: Alec Radford Karthik Narasimhan Tim Salimans Ilya Sutskever
references.title: Improving Language Understanding by Generative Pre-Training
references.venue: Techincal Report,
references.authors: Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei Ilya Sutskever
references.title: Language Models are Unsupervised Multitask Learners
references.venue: OpenAI Blog,
references.authors: Mohammad Rastegari Vicente Ordonez Joseph Redmon Ali Farhadi
references.title: XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
references.venue: In ECCV,
references.authors: Sashank J Reddi Satyen Kale Sanjiv Kumar
references.title: On the Convergence of Adam and Beyond
references.venue: In ICLR,
references.authors: Russell Reed
references.title: Pruning Algorithms-A Survey
references.venue: IEEE Transactions on Neural Networks,
references.authors: Florian Schroff Dmitry Kalenichenko James Philbin
references.title: FaceNet: A Unified Embedding for Face Recognition and Clustering
references.venue: In CVPR,
references.authors: Gunnar A Sigurdsson Gül Varol Xiaolong Wang Ali Farhadi Ivan Laptev Abhinav Gupta
references.title: Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
references.authors: Gunnar A Sigurdsson Santosh Divvala Ali Farhadi Abhinav Gupta
references.title: Asynchronous Temporal Fields for Action Recognition
references.authors: Khurram Soomro Amir Roshan Zamir Mubarak Shah
references.title: UCF101: A Dataset of 101 Human Action Classes
references.venue: From Videos in The Wild. CRCV-TR-12-01,
references.authors: Nitish Srivastava Russ R Salakhutdinov
references.title: Multimodal Learning with Deep Boltzmann Machines
references.venue: In NeurIPS,
references.authors: Weijie Su Xizhou Zhu Yue Cao Bin Li Lewei Lu Furu Wei Jifeng Dai
references.title: VL-BERT: Pre-training of Generic Visual-Linguistic Representations
references.venue: In ICLR,
references.authors: Chen Sun Fabien Baradel Kevin Murphy Cordelia Schmid
references.title: Learning Video Representations using Contrastive Bidirectional Transformer
references.venue: arXiv preprint arXiv:1906.05743,
references.authors: Chen Sun Austin Myers Carl Vondrick Kevin Murphy Cordelia Schmid
references.title: VideoBERT: A Joint Model for Video and Language Representation Learning
references.venue: In ICCV,
references.authors: Hao Tan Mohit Bansal
references.title: LXMERT: Learning Cross-Modality Encoder Representations from Transformers
references.venue: In EMNLP-IJCNLP,
references.authors: Yapeng Tian Dingzeyu Li Chenliang Xu
references.title: Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing
references.venue: In ECCV,
references.authors: Ashish Vaswani Noam Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan N Gomez Łukasz Kaiser Illia Polosukhin
references.title: Attention Is All You Need
references.venue: NeurIPS,
references.authors: Mike Wu Chengxu Zhuang Milan Mosse Daniel Yamins Noah Goodman
references.title: On Mutual Information in Contrastive Learning for Visual Representations
references.venue: arXiv preprint arXiv:2005.13149,
references.authors: Dejing Xu Jun Xiao Zhou Zhao Jian Shao Di Xie Yueting Zhuang
references.title: Self-supervised Spatiotemporal Learning via Video Clip Order Prediction
references.authors: Zhilin Yang Zihang Dai Yiming Yang Jaime Carbonell Russ R Salakhutdinov Quoc V Le
references.title: XLNet: Generalized Autoregressive Pretraining for Language Understanding
references.venue: NeurIPS,
references.authors: Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun
references.title: ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
references.authors: Hang Zhao Chuang Gan Andrew Rouditchenko Carl Vondrick Josh McDermott Antonio Torralba
references.title: The Sound of Pixels
references.venue: In ECCV,
references.authors: Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J Corso Jianfeng Gao
references.title: Unified Vision-Language Pre-Training for Image Captioning and VQA
references.venue: In AAAI,
references.authors: Zisserman
references.title: For the audio CNN, we use a ResNet-50 without the downsampling layer pool1 to preserve information along both frequency and time axis in early stages
sections.heading: 1 INTRODUCTION
sections.text: Learning multimodal representation from unlabeled videos has received considerable attention (Baltrušaitis et al., 2018). Audio-visual learning is of particular interest due to the abundance of videos with natural audio-visual co-occurrence (Owens & Efros, 2018; Owens et al., 2018; Arandjelovic & Zisserman, 2018; Ephrat et al., 2018; Gao & Grauman, 2019; Alwassel et al., 2019). However, existing approaches learn localized representations from short videos (hundreds of milliseconds to just under a few seconds), capturing only short-term dependencies in data. While this is useful for certain applications, e.g., source separation (Ephrat et al., 2018) and atomic action recognition (Gu et al., 2018), learning representation that captures long-term dependencies is equally important, e.g., for activity recognition (Kay et al., 2017; Carreira et al., 2019; Sigurdsson et al., 2016). Unfortunately, processing long videos requires large memory resource and capturing long-term dependencies is a long-standing problem (Hochreiter & Schmidhuber, 1997; Cho et al., 2014; Vaswani et al., 2017).
  In language understanding, strong progress has been made in large-scale learning of contextualized language representations using Transformers (Vaswani et al., 2017; Howard & Ruder, 2018; Peters et al., 2018; Radford et al., 2018; 2019; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019). Riding on the success of Transformers, several recent works have extended it to the multimodal setting by adding an additional vision module to the Transformer framework (Sun et al., 2019b; Lu et al., 2019). However, these models are typically not end-to-end trained; they rely on a language-pretrained BERT (Devlin et al., 2019), which is fixed throughout, and train only the visual components. While the pretrained BERT helps accelerate convergence and brings reliable extra supervision signal to the
  vision component, this partial learning setup can be undesirable if the text data comes from different distributions (of topics, dialects, or foreign languages) or if we want to apply it to different modalities (e.g., audio-visual). Unfortunately, end-to-end training of such multimodal Transformer architectures is challenging for most existing compute environments due to the excessive memory requirement.
  In this work, we make three key contributions. First, we propose an end-to-end trainable bidirectional transformer architecture that learns contextualized audio-visual representations of long videos. Our model, shown in Figure 1, consists of audio/visual CNNs, audio/visual Transformers, and a multimodal Transformer. The CNNs operate on short (e.g., one second) video clips and are intended to capture short-term dynamics within each modality. The Transformer layers operate on long video sequences (e.g., 30 seconds), capturing long-term dynamics. To enable end-to-end training, we propose a novel parameter reduction technique that shares parts of weight parameters across Transformers and across layers within each Transformer. We show that this results in up to 97% parameter reduction, enabling end-to-end training of our model, with a minimal performance degradation. To the best of our knowledge, our work is the first to report end-to-end trained multimodal Transformers, and the first to apply Transformers for audio-visual representation learning.
  The quality of negative samples is crucial in contrastive learning, which is part of our learning objective. As our second contribution, we propose a content-aware negative sampling strategy that favors negatives sufficiently similar to a positive instance. Our approach measures the similarity by reusing the CNN embeddings obtained during model training, and thus do not introduce extra parameters to learn. We show that this improves performance over the standard sampling strategies.
  Our third contribution is a systematic evaluation of different modality fusion strategies. Existing works on multimodal BERT (all using vision-and-language data) typically apply one fusion strategy without thoroughly comparing with alternatives, e.g., some works perform early fusion (Sun et al., 2019b; Su et al., 2020) while others perform mid-level fusion (Lu et al., 2019; Tan & Bansal, 2019). As a result, it is unclear how different fusion methods affect the final performance. In this work, we compare three fusion strategies (early, mid, late) and show the superiority of mid-level fusion.
  To demonstrate our approach, we pretrain our model on long (30-second) video clips from Kinetics700 (Carreira et al., 2019) and finetune it on various video classification tasks. One benefit of the modular design of our architecture is flexibility: once pretrained, we can use any of the subnetworks for downstream tasks depending on the modalities involved (audio-only, visual-only, audio-visual) and video lengths (short and long). To show this, we evaluate our model on UCF101 (Soomro et al., 2012) and ESC-50 (Gemmeke et al., 2017) for short-term visual/audio classification, and Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017) for long-term audio-visual action recognition.
sections.heading: 2 APPROACH
sections.text: Figure 1 shows an overview of the proposed model architecture. The input to our model is a sequence of visual clips v1:T and the corresponding sequence of audio streams a1:T . For example, each
  sequence is a 30 second-long video divided into 30 non-overlapping clips (each clip is one second long). We divide our model into three parts with different characteristics, which are explained below.
  Local Feature Embedding. We feed each of T video clips to a visual CNN fV (vt) to obtain xv1:T ∈ RT×D, and each audio stream to an audio CNN fA(at) to obtain xa1:T ∈ RT×D.1 Intuitively, the CNN outputs are temporally local embeddings as they have access to only a short-range temporal window of the entire video sequence. Thus, they are suitable for representing short-range atomic actions (e.g., sit down, raise arms) that constitute long-range events (e.g., gym workout). We use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016) as a visual CNN fV , and a ResNet-50 as an audio CNN fA. The weights of both CNNs are randomly initialized and trained end-to-end with the Transformer layers.
  Unimodal Contextualized Embedding. The local feature embeddings capture short-term dynamics but lack long-term contextual information. We use Transformers (Vaswani et al., 2017) to enrich the embeddings with sequence-level context. We start by learning unimodal contextualized representations using the visual Transformer gV and the audio Transformer gA, respectively.
  The Transformer consists of L layers, each with two sub-layers: a multi-head attention layer and a feed-forward layer. Given an input sequence of embeddings x ∈ RT×D and A attention heads, the j-th head in the attention layer computes the output embedding sequence aj ∈ RT×γ , γ = D/A as
  aj = softmax
  ( QjK
  > j√ γ
  ) Vj , Qj = xW q j ,Kj = xW k j , Vj = xW v j (1)
  where W qj ,W k j ,W v j ∈ RD×γ are weight matrices for computing the (query, key, value) triplet given the input x. This operation is repeated for each attention head, and the outputs are combined (with concatenation followed by one linear layer with weights W b ∈ RD×D), producing a ∈ RT×D. Next, the feed-forward layer takes this intermediate output and computes o ∈ RT×D using a twolayer fully-connected network with weights W c ∈ RD×E and W d ∈ RE×D. The output of each sub-layer is computed using a residual function followed by layer normalization (Ba et al., 2016), i.e., LayerNorm(x+ Sublayer(x)). In this work, we set the number of layers L = 6, the number of attention heads A = 12, the feature dimension D = 768 and the intermediate dimension E = 3072. For simplicity, we use this design for all layers across all three Transformers in our model.
  Before feeding local embeddings xv and xa to unimodal Transformers, we augment them with “positional” embeddings. Specifically, we append to the beginning of each sequence a special vector BOS (beginning of sequence), i.e., xv0 for visual and x a 0 for audio streams; their dimensions are same as xvt and x a t , respectively. We also define positional embeddings p0:T encoding time indices (we call this “time” embedding). This is necessary to preserve information about temporal ordering of local feature embeddings, which is otherwise lost in Eqn. 1. We combine them via layer normalization,
  uvt = LayerNorm(x v t + p v t ), u a t = LayerNorm(x a t + p a t ), ∀t ∈ [0, T ] (2)
  We initialize {xv0,xa0 ,pv0:T ,pa0:T } to the normal distribution and train them with the rest of the model. We feed the augmented visual embeddings into the visual Transformer gV and obtain yv0:T = gV (u v 0:T ), and similarly obtain y a 0:T = gA(u a 0:T ). The embeddings at each time step has a direct access to the entire input sequence regardless of their position (it has a one-step signal path during forward and backward inference). Multiple layers of such feature transformation thus allow the resulting embedding to be deeply contextualized in the time dimension. We denote the output embeddings corresponding to the BOS positions by BOSvg = y v 0 and BOS a g = y a 0 , and designate them as the summary embeddings representing the sequence of each modality.
  Multimodal Contextualized Embedding. The unimodal embeddings capture long-term temporal context but miss out on cross-modal information. The final step in forward inference is to use a multimodal Transformer hAV to obtain embeddings contextualized in the audio-visual space.
  We first augment the embeddings yv0:T and y a 0:T with modality and time embeddings. The modality embeddings mv and ma are vectors of the same dimension as yvt and y a t , respectively. We share m v (and ma) across all the unimodal embeddings yv0:T (and y a 0:T ); thus, they add modality-discriminative information to the Transformer. We also add time embeddings p0:T as before; however, unlike in
  1For notational simplicity, we drop the subscripts to refer to the entire sequence unless distinction is necessary.
  the previous step, we share the same p0:T between embeddings from the two modalities to correctly indicate the time indices. We augment the modality and time embeddings via layer normalization,
  wvt = LayerNorm(y v t + pt + m v), wat = LayerNorm(y a t + pt + m a), ∀t ∈ [0, T ] (3)
  We feed the augmented visual embeddings wv0:T and audio embeddings w a 0:T to the multimodal Transformer hAV , one after another, and obtain z0:(2T+1) = hAV ([wv0:T ;w a 0:T ]). We again denote the output embeddings corresponding to the BOS positions by BOSvh = z v 0(= z0) and BOS a h = z a 0(= zT+1), and use them as summary embeddings encoding multimodal context.
  We emphasize the importance of feeding wv0:T and w a 0:T one after another. An alternative would be concatenating them before feeding them to hAV and obtaining an output z0:T (instead of z0:(2T+1)). However, this restricts the Transformer to access audio-visual embeddings only from the same time slices, which could be problematic when there is a temporally asynchronous relationship between the two modalities (e.g., a visual clip matches with sound captured a few times steps before) (Kazakos et al., 2019; Morgado et al., 2020). By arranging the two sequences one after the other, the Transformer can mix-and-match appropriate audio-visual embeddings in an asynchronous manner. Another practical concern with the alternative approach is that it significantly increases the model size; the weight matrices Wq,Wk,Wv grow quadratically with the input feature dimension D. Serializing the input resolves both issues.
sections.heading: 2.1 SELF-SUPERVISED PRETRAINING OBJECTIVES
sections.text: Task 1: Masked Embedding Prediction (MEP). BERT (Devlin et al., 2019) is trained using the masked language model (MLM) task, which randomly selects input tokens and replaces them with a mask token. The model is then trained to predict the original (unmasked) tokens by solving a classification task with a cross-entropy loss. However, inputs to our model are real-valued audiovisual signals (rather than discrete tokens),2 so applying the MLM task requires input discretization, which causes information loss (Lu et al., 2019; Sun et al., 2019a). We instead train our model to identify the correct visual clip or audio stream compared to a set of negative samples in a contrastive manner, which does not require input discretization.
  We formulate our MEP task using InfoNCE (Oord et al., 2018), which is the softmax version of the noise contrastive estimation (NCE) (Gutmann & Hyvärinen, 2010). Let õt be the t-th output of any of the three Transformers obtained by masking the t-th input xt. Our InfoNCE loss is then defined as
  LNCE(x, õ) = −Ex [∑ t log I(xt, õt) I(xt, õt) + ∑ j∈neg(t) I(xj , õt) ] , (4)
  where neg(t) are negative sample indices and the compatibility function I(xt, õt) is, I(xt, õt) = exp ( FFN>(õt)WIxt ) , (5)
  whereWI ∈ RP×D (P = 256) and FFN is a two-layer feed-forward network. The use of a non-linear prediction head has shown to improve the quality of the representations learned in a contrastive learning setup (Chen et al., 2020); following the recent work in Transformers (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020), we use a GELU non-linear activation function (Hendrycks & Gimpel, 2016) in FFN. Optimizing Eqn. 4 enforces I(xt, õt) to approximate the density ratio
  p(xt|õt) p(xt)
  ; this can be seen as maximizing the mutual information between xt and õt (Oord et al., 2018). Intuitively, this encourages the Transformer to capture the underlying dynamics of x from each modality without explicitly learning a generative model p(xt|õt). Negative sampling. We find that a good negative sampling strategy is essential for the model’s convergence. Existing approaches either use all but xt (positive) within a mini-batch as negative samples or limit it to the current sequence only. However, both these methods ignore the data content and thus can miss useful negatives. Oord et al. (2018) showed that leveraging prior knowledge about data can improve the negative sample quality (e.g., by sampling negatives from the same speaker as the positive). Unfortunately, such prior knowledge is often not available in unlabeled videos.
  2In the form of RGB images and log-mel-scaled spectrograms.
  We propose a content-aware negative sampling strategy that favors negatives sufficiently similar to a positive instance in the CNN embedding space; we call our approach CANS-Similar. Our approach is inspired by Ulyanov et al. (2018) who showed that randomly initialized CNNs provide a strong prior over natural images due to the inductive bias already built into the design of the CNNs. This suggests that our local feature embeddings xv (and xa) can capture the underlying statistical regularities in video clips (and audio streams) right from the beginning, which can be sufficient to assess the similarity/dissimilarity between clips. Therefore, the distance measured on them can approximate content dissimilarity well (and this will improve as the training progresses).
  Motivated by this, we sample the negatives based on local feature embeddings xv (and xa). Specifically, we compute a pairwise `2 distance between xt (positive) and all other instances within a mini-batch, and normalize them to the [0, 1] interval. To remove samples that are either too similar or too different from the positive sample, we discard instances that fall outside the 95% confidence interval in the normalized distance space. We then sample the negatives from the remainder using the normalized distance as sampling probability. This makes instances similar to the positive instance have more chance to become negatives. We emphasize the importance of sampling, instead of deterministically taking top most similar samples; the stochasticity allows our model to be robust to potentially inaccurate distance estimates because samples with low probabilities will still have a chance to be selected as negatives.
  Finally, our MEP loss is the InfoNCE loss computed on all three Transformers, LMEP = LNCE(xa, ỹa) + LNCE(xv, ỹv) + LNCE([xa;xv], z̃) (6)
  Task 2: Correct Pair Prediction (CPP). The MEP task encourages our model to learn the underlying dynamics within each modality. To help our model learn cross-modal dynamics, we design a task that predicts whether a pair of audio-visual embeddings is from the same video. Specifically, we define two binary classifiers, one for the two unimodal Transformers and another for the multimodal Transformer. Each classifier takes as input either sg = [yv0 ;y a 0 ] (and [z v 0; z a 0 ]), a pair of audio-visual “summary” embeddings corresponding to the BOS positions, or sh = [yvt ;y a t ] (or [z v t ; z a t ]), the output embeddings sampled at random positions (we take two random positions t ∈ [1, T ]). The classifier predicts p(c|s) indicating whether the pair is from the same video (c = 1) or from different videos (c = 0). We train the classifiers with a binary cross-entropy loss,
  LCPP = −Ex,y [c · log p(c|sg) + c · log p(c|sh)] (7) where · is the inner product. We generate a random derangement of the input mini-batch so that the number of positive and negative pairs are guaranteed to be the same.
  Overall Pretraining Objective. We train our model end-to-end from scratch by optimizing LMEP + αLCPP with a balancing term α. We find our model is insensitive to this term, so we set α = 1.0.
sections.heading: 2.2 PARAMETER REDUCTION
sections.text: Optimizing our model is challenging due to the large memory requirement. The most expensive part is the Transformers, which take up 82% of model parameters. One could reduce the model size by making the Transformers shallower, but the depth of Transformers has shown to be crucial to get good performance (Devlin et al., 2019). We propose to reduce the model size by aggressively sharing parts of weights across Transformers as well as layers within each Transformer (see Figure 2 (d)).
  Sharing across Transformers. We first consider sharing weights across Transformers. Each Transformer encodes data coming from different distributions: gV encodes xv, gA encodes xa, and
  hAV encodes (yv,ya). These input distributions may each exhibit different dynamics, yet together share certain regularities because they all come from the same videos. Motivated by this, we decompose Transformer weights into shared and private parts so that different patterns can be learned in a parameter-efficient manner. Recall that each layer of a Transformer contains weights {W q,W k,W v,W b,W c,W d}. We decompose each of these weights into W = UΣV >, where W ∈ RM×N , U ∈ RM×O,Σ ∈ RO×O, V ∈ RN×O. We perform low-rank approximation of W by setting the rank O M,N , and share U across Transformers while keeping Σ and V private to each Transformer. This helps reduce parameters becauseMO+3(O2 +NO) 3MN . We experimented with different matrix ranks O but the differences were small; we set O = 128 (M,N = 768 or 3072).
  The decomposition converts a linear projection of input Wx into a series of (unconstrained) linear projections UΣV >x. However, this can cause numerical instability during optimization (Nocedal & Wright, 2006). We could perform the Singular Value Decomposition (SVD) over W so that it performs rotation (V >), stretch (Σ), and rotation (U ) with orthogonal basis vectors in U and V . Unfortunately, solving the full SVD has a computational complexity of O(max(M,N)2) (Golub & Van Loan, 2012). Here, we put an orthogonality constraint only on Σ and perform projection (V >), rotation (Σ), and projection (U ) of input x. In addition, we put V >x in a unit sphere (via `2- normalization) before rotating it with Σ. This not only improves numerical stability, but also removes magnitude information in V >x and keeps angular information only, which has been shown to provide sample discriminative information (Chen et al., 2019a). To impose the orthogonality constraint on Σ, we use the Padé approximation with a scale-squaring trick of Lezcano-Casado & Martínez-Rubio (2019). Intuitively, we linearly project x onto a unit sphere (V >x) and rotate it (ΣV >x) in each Transformer so that it captures the dynamics of each input distribution independently. We then project it to the shared space via U , capturing shared regularities across all three Transformers.
  Sharing across Layers. Recently, Bai et al. (2019a) showed that sharing parameters across layers in deep neural networks does not hurt the representational power of the network. Furthermore, (Lan et al., 2020) demonstrated that cross-layer parameter sharing in the Transformer leads to a lighter and faster-to-train model without sacrificing the performance on various language understanding benchmarks. Motivated by this, we let each Transformer share parameters across different layers.
sections.heading: 3 EXPERIMENTS
sections.text: We pretrain our model on Kinetics-700 (Carreira et al., 2019) or AudioSet (Gemmeke et al., 2017) and finetune it on various downstream tasks. The official release of Kinetics-700 contains 10-second clips only, so we download 410K original videos from YouTube and take 30-second clips from each video. For fair comparison with prior work, we use 10-second clips from the official release of AudioSet (we used 1.8M clips). We pretrain our model on 64 NVIDIA Tesla V100 GPUs with a batch size of 256 for 220K iterations. For downstream tasks, we evaluate on short-video/audio classification using UCF-101 (Soomro et al., 2012) (13K clips from 101 classes; 7.2 seconds on average) and ESC50 (Gemmeke et al., 2017) (2K clips from 50 classes; 5 seconds), and on long-video classification using Kinetics-Sounds (Arandjelovic & Zisserman, 2017) (23K videos from 32 classes; 10 seconds on average) and Charades (Sigurdsson et al., 2016) (10K videos from 157 classes; 30 seconds on average). We describe various details about experimental setup in Appendix.
sections.heading: 3.1 RESULTS AND DISCUSSION
sections.text: Multimodal Fusion Methods. To evaluate different fusion methods on the quality of learned representation, we test the following settings: (i) Early uses a single multimodal Transformer with 2 × L layers, (ii) Mid is our approach described in Figure 1, (iii) Late uses two unimodal Transformers each with 2× L layers. All the methods are pretrained on audio-visual data using CPP and MEP losses, except for (iv) Late-w/o-CPP where we use only the MEP loss. We finetune the pretrained models on audio-visual, audio-only, and visual-only scenarios. For fair comparisons across different fusion methods, we do not perform parameter sharing in this ablation setting.
  Table 1 (a) shows that Early and Mid outperform Late on the audio-visual scenario. This suggests the importance of encoding cross-modal information. Note that Late-w/-CPP gets cross-modal self-supervision, which gives marginal performance improvement over Late-w/o-CPP; however, both methods miss the opportunity to encode any cross-modal relationship, leading to inferior results.
  While both Early and Late perform similarly in the audio-visual scenario, only Late can be used in unimodal downstream scenarios (c.f., Early requires the presence of both modalities). This has practical implications: Mid and Late can effectively handle missing modalities, i.e., once pretrained on audio-visual data, we can use it on any of audio-visual, audio-only, and visual-only scenarios. Our Mid fusion approach enjoys both the advantages, i.e., learning cross-modal relationship and being robust to missing modalities, achieving overall the best performance.
  Negative Sampling Strategies. We compare four strategies: (i) Current-Sequence takes all but the positive instance from the same sequence as negatives, (ii) Current-MiniBatch takes all but the positive instance in the mini-batch as negatives; this subsumes Current-Sequence, (iii) CANS-Dissimilar stochastically samples negatives using a modified version of our contentaware negative sampling (CANS) that favors dissimilar samples, and (iv) CANS-Similar is our proposed CANS approach that favors negatives that are similar to the positive instance.
  Table 1 (b) shows Current-Sequence is the least effective: It makes MEP too difficult because negatives are (sometimes too much) similar to positives. As a result, the training dynamics is dominated by CPP, which is relatively easier, leading to inferior performance. We make quite the contrary observations from Current-MiniBatch: the inclusion of negatives from different videos makes MEP easier and thus makes it dominate the training dynamics. Our CANS approach solves both these issues by eliminating negatives that are either almost identical to or trivial to distinguish from the positives, based on the 95% CI over the CNN embedding distances. It also samples negatives in a stochastic manner so a wide variety of samples can be included as negatives. Our proposed CANS-Similar can be considered as a “softened” version of Current-Sequence; it samples negatives that are similar to positives with a high probability (this can be considered as online hard negative mining), but it also takes instances from different videos with a lower probability. This balances out hard and easy negatives, making the MEP task effective.
  Parameter Sharing Schemes. Our parameter reduction scheme reduces the number of parameters from 128M to 4M (by 97%) (Table 1 (c)). We reduce the model size by sharing weights across Transformers and across layers. We validate these ideas in two sets of experiments. Table 1 (c) compares cross-Transformer weight sharing schemes. We use Multi-6 that uses all three Transformers with 6 layers each, and compare four methods that correspond to Figure 2 (a)-(d). Note that No sharing is too large to fit in a Tesla V100 GPU (16GB) even with 2 samples, so we define Multi-2 that uses three Transformers with 2 layers each, and with the reduced number of attention heads A to 5, the feature dimension D to 320 and the intermediate dimension E to 1280. We see that our proposed approach, Part, achieves the best performance with the least number of parameters. One might ask how Part leads to a smaller model when All shares all the weights across Transformers: We decompose weights W = UΣV > with low-rank approximation and share only U across Transformers, while the ΣV > part learns modality-specific dynamics. Table 1 (d) compares cross-layer weight sharing schemes using the visual Transformer with either 2 (Vis-2) or 6 (Vis-6) layers. The results show that sharing weights across layers does not hurt the performance, confirming the observations by Lan et al. (2020) in the audio-visual setting.
  Pretraining Objectives. To evaluate the importance of MEP and CPP tasks, we test two settings: (i) Mid-w/o-CPP and (ii) Mid-w/o-MEP. On Kinetics-Sounds, these achieve 65.9% and 64.6%, respectively; ours achieve 67.5% (top-1 accuracy). The result show that the MEP task plays an important role during pretraining, confirming the findings from Sun et al. (2019a) that the InfoNCE
  a) Model Net Data UCF ST-Puzzle 3D-R18 K400 65.8 ClipOrder R(2+1)D UCF 72.4 DPC 3D-R34 K400 75.7 CBT S3D K600 79.5 MultiSens 3D-R18 AS 82.1 AVTS MC3-18 K400 85.8 AVTS MC3-18 AS 89.0 V-CNN† SlowFast K700 85.2 V-CNN† SlowFast AS 86.1
  b) Model Net Data ESC SVM MLP - 39.6 ConvAE CNN-4 39.9 RF MLP - 44.3 ConvNet CNN-4 - 64.5 SoundNet CNN-8 FS 74.2 L3-Net CNN-8 FS 79.3 DMC VGG-ish FS 79.8 AVTS VGG-M AS 80.6 A-CNN† R50 AS 81.5
  c) Model Charades KS Random 5.9 - / - ATF 18.3 - / - ATF (OF) 22.4 - / - V-CNN 18.7 45.8 / 73.3 A-CNN 18.9 49.4 / 76.9 M-CNN 23.1 59.4 / 83.6 V-BERT 26.0 49.5 / 78.9 A-BERT 27.4 58.9 / 85.7 M-BERT† 29.5 75.6 / 94.6
  Datasets. K: Kinetics, AS: AudioSet, FS: Flicker-SoundNet, KS: Kinetics-Sounds. Baselines. ST-Puzzle (Kim et al., 2019), ClipOrder (Xu et al., 2019), DPC (Han et al., 2019), CBT (Sun et al., 2019a), MultiSens (Owens & Efros, 2018), AVTS (Korbar et al., 2018), AE (Aytar et al., 2016), SVM (Piczak, 2015a), RF (Piczak, 2015a), ConvNet (Piczak, 2015b), SoundNet (Aytar et al., 2016), L3-Net (Arandjelovic & Zisserman, 2017), DMC (Hu et al., 2019), ATF (Sigurdsson et al., 2017)
  Table 2: (a; left): Short video classification results on UCF101 (mean accuracy (%)). (b; center): Short audio classification results on ESC-50 (mean accuracy (%)). (c; right): Long video classification results on Charades (mAP) and Kinetics-Sounds (KS; top-1/5 accuracy (%)). †: Ours.
  loss, as deployed in CBT, is effective in the cross-modal setting. The result also shows that augmenting MEP with CPP provides further performance improvement by learning cross-modal correspondence.
  Downstream Evaluation. We pretrain our model with Mid fusion using MEP and CPP tasks (with CANS-Similar), and employ Part weight sharing. We use either Kinetics-700 or AudioSet for fair comparisons with prior work. Table 2 (a)/(b) shows short-video/audio classification results on UCF-101/ESC-50. For fair comparisons to the baselines, we use only the visual/audio CNN (no Transformers); we finetune a linear classifier on top of the visual CNN end-to-end for UCF-101, and train a multi-class one-vs-all linear SVM on top of the fixed audio CNN for ESC-50. Although our model is pretrained on long video clips with no direct supervision to the CNN layers (gradients must flow through Transformers), it outperforms most of the baselines (except for AVTS on UCF-101) that received direct supervision from short video clips. We note that, similar to ours, CBT (Su et al., 2020) is a multimodal Transformer pretrained on long video clips and thus is the most meaningful comparison to ours; ours outperform CBT on UCF-101 by 5.7%. For sound classification, our approach outperform all existing published results.
  Table 2 (c) shows long-video classification results on Charades and Kinetics-Sounds (KS) when pretrained on Kinetics-700. We test Visual-only (V), Audio-only (A), and Multimodal (M) settings to verify the benefit of multimodal learning. Because there is no published selfsupervised learning results on these datasets, we demonstrate long-term representations by comparing CNNs (CNN; short-term) to Transformers (BERT; long-term) on KS that contains 10-second clips. Since CNNs process 1-second clips, we feed 10 non-overlapping clips to CNNs and average the prediction output. In all settings, we add a 2-layer MLP with softmax classifier on top. The results show that Transformers outperform CNNs on Kinetics-Sounds, suggesting the superiority of long-term representations. We also see that combining audio-visual information performs the best. We notice that audio representations are generally stronger than visual representations; we believe that learning discriminative visual representations is generally more challenging, especially when the CNNs receive (self-)supervision signals only indirectly through Transformers. We believe that providing (self-)supervision directly to CNNs, e.g., by first pretraining CNNs on 3D rotation prediction (Jing et al., 2018) and then jointly training the whole model (as was done in CBT (Sun et al., 2019a)), could further improve performance. Incorporating contrastive learning (Chen et al., 2020) over the CNN embeddings and training the whole model end-to-end is another promising direction for future work.
sections.heading: 4 RELATED WORK
sections.text: Multimodal BERT. Extending BERT (Devlin et al., 2019) to vision-and-language has been actively studied. Existing work typically adopt early fusion (Li et al., 2019; Alberti et al., 2019; Sun et al., 2019b; Li et al., 2020; Zhou et al., 2020; Su et al., 2020; Chen et al., 2019b; Zhu & Yang, 2020) or mid fusion (Tan & Bansal, 2019; Lu et al., 2019; Sun et al., 2019a; Luo et al., 2020) without thorough validation, and they train only visual components while relying on a language-pretrained BERT. Although there have been some efforts to leverage the Transformer architecture (Vaswani et al., 2017) for audio and visual inputs (Boes & Van hamme, 2019; Tian et al., 2020), our approach is the first to demonstrate multimodal audio-visual BERT trained from scratch in an end-to-end manner. This is enabled by our novel parameter reduction technique, which is one of our main technical contributions.
  Audio-Visual Learning. Early work in audio-visual learning focused on speech signals, improving audio-visual speech recognition than unimodal approaches (Ngiam et al., 2011; Srivastava & Salakhutdinov, 2012). Recent approaches leverage unlabeled videos from specific domains (Owens et al., 2016; Gao & Grauman, 2019; Zhao et al., 2018; Ephrat et al., 2018; Alwassel et al., 2019; Miech et al., 2020; Piergiovanni et al., 2020) and often demonstrate on audio-visual source separation, localization, and co-segmentation. However, these approaches rely on short-term audio-visual correspondence and thus may not generalize to long-term video recognition that requires global context (as was suggested in (Hjelm et al., 2019)), which this work focuses on.
  Parameter Reduction. Network pruning (Reed, 1993; Caron et al., 2020) trains a large model and then reduces its size while maintaining performance. Reducing the size of CNNs for mobile applications is an active research area (Rastegari et al., 2016; Howard et al., 2017; 2019; Zhang et al., 2018; Iandola et al., 2016). Our work is closely related to the work that shares parameters across layers in deep neural networks. Trellis network (Bai et al., 2019b) is a temporal convolutional architecture with weight-tying across time and depth. Similar to ours, Universal Transformer (Dehghani et al., 2019), RSNMT (Dabre & Fujita, 2019), DEQ (Bai et al., 2019a), ALBERT (Lan et al., 2020) share weights across layers in Transformers. We combine this idea with our novel cross-Transformer weight sharing, which decomposes weight matrices with low-rank approximation.
  Negative Sampling. Hard negative mining has been shown to be crucial for contrastive learning (Arandjelovic & Zisserman, 2017; Owens & Efros, 2018; Korbar et al., 2018; Schroff et al., 2015; Zhuang et al., 2019; Morgado et al., 2020; Wu et al., 2020). Korbar et al. (2018) use the time difference between clips to approximate clip similarity (i.e., clips that are further apart are deemed more different). However, such an assumption may not hold for real-world videos, e.g., periodic actions such as push-ups. Unlike this line of approaches, we directly use the feature embeddings learned by our model. Several apparoaches adapted a similar idea (Schroff et al., 2015; Zhuang et al., 2019; Morgado et al., 2020; Wu et al., 2020). Different from prior work, we bring the stochasticity to the sampling procedure by using the content similarity as the sampling probability; this helps reduce potential errors especially during the early stage of training.
sections.heading: 5 CONCLUSION
sections.text: We introduced a multimodal bidirectional Transformer architecture for self-supervised learning of contextualized audio-visual representation from unlabeled videos. Our main technical contributions include: (1) we propose a parameter efficient multimodal Transformers based on matrix decomposition with low-rank approximation; (2) we propose a novel content-aware negative sampling technique for contrastive learning. We demonstrate a successful end-to-end training of multimodal Transformers for audio-visual learning (which is, to the best of our knowledge, the first time in the literature). We also report comprehensive evaluation of various design decisions in multimodal learning.
  Acknowledgements. This work was partially supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2017-0-01772, Video Turing Test, No.2019-0-01082, SW StarLab) and the international cooperation program by the NRF of Korea (NRF-2018K2A9A2A11080927).
sections.heading: A.1 ARCHITECTURES OF VISUAL/AUDIO CNNS
sections.text: Table 3 shows the architectures of visual and audio CNNs we use for our model. For the visual CNN, we use the SlowFast network (Feichtenhofer et al., 2019) with a ResNet-50 backbone (He et al., 2016). We use the speed ratio α = 8 and the channel ratio β = 1/8 for the SlowFast architecture, so Tf = 8 × Ts. We use different values of Ts and Tf for different tasks. During pretraining, we set Ts = 4 and Tf = 32. During finetuning, we use Ts = 8 and Tf = 64 for short-video action classification on UCF101 (Soomro et al., 2012) while we use Ts = 4 and Tf = 32 for long-video action classification on Charades (Sigurdsson et al., 2016) and Kinetics-Sounds (Arandjelovic & Zisserman, 2017). For the audio CNN, we use a ResNet-50 without the downsampling layer pool1 to preserve information along both frequency and time axis in early stages. We use different values of Ta for different training phases. We set Ta = 220 for one-second clip during pretraining while we use Ta = 440 for two-second clip during finetuning.
sections.heading: A.2 DATA PREPROCESSING
sections.text: We preprocess the data by dividing T -second clips into T non-overlapping parts (T = 30 for Kinetics700 (Carreira et al., 2019) and T = 10 for AudioSet (Gemmeke et al., 2017)) and sampling 16 frames from each. For audio stream, we take waveform sampled at 44.1 kHz and convert it to log-mel-scaled spectrogram. We augment audio data with random frequency/time masking using SpecAugment (Park et al., 2019), and visual data with color normalization, random resizing, random horizontal flip, and
  random cropping to obtain 112× 112 pixel frames; for test data, we resize videos to 128 pixels on the shorter side and take three equidistant crops of 128× 128 pixels to cover the entire region. We also apply audio-visual synchronized temporal jittering (Patrick et al., 2020).
sections.heading: A.3 DOWNSTREAM EVALUATION
sections.text: For evaluation on UCF101, we follow the test protocol of (Feichtenhofer et al., 2019): We sample 10 clips from each test video at a uniform time interval, and for each sampled clip, we take three equidistant spatial crops, resulting in a total of 30 views. We use each of the 30 views as input to our visual CNN and average the prediction scores from all 30 views to obtain the final prediction result. For evaluation on ESC-50 (Piczak, 2015a), we extract 10 equally spaced 2-second clips from each test audio sample. We use each of 10 clips as input to our audio CNN and average the prediction scores to obtain the final prediction result. For evaluation on Charades and Kinetics-Sounds, we use three audio-visual sequences with different spatial crops from a test video and max-pool/average the prediction scores from each sequence, respectively.
sections.heading: A.4 OPTIMIZATION
sections.text: In all experiments, we use the AMSGrad (Reddi et al., 2018) variant of AdamW (Loshchilov & Hutter, 2019) optimizer with β1 = 0.9, β2 = 0.98, L2 weight decay of 1e-4. We use a learning rate warm-up for the first 6% of iterations followed by a linear decay of learning rate.
  From the observations of Lezcano-Casado and Martínez-Rubio (Lezcano-Casado & Martínez-Rubio, 2019), we have 10 times less learning rate for the orthogonal parameters than that for the nonorthogonal parameters: we use 1e-5 for the former and 1e-4 for the latter.
  We pretrain our model on Kinetics-700 (Carreira et al., 2019) with a batch size 256 for 220K iterations and AudioSet (Gemmeke et al., 2017) with a batch size 300 for 220K iterations in the main experiments; for the ablation study, we use a much smaller batch size of 4 and pretrain our model on Kinetics-700 for 80K iterations.
  For finetuning on UCF101, we train our model for 40K iterations with a batch size of 64 and learning rate of 0.02. For evaluation on ESC-50, we train a multi-class one-vs-all linear SVM on top of our fixed audio CNN for 38K iterations with a batch size of 128 and learning rate of 0.003. For finetuning on Charades, we train for 40K iterations with a batch size of 8, with learning rate of 0.001 for the classifier and CNN parameters, 1e-5 for the orthogonal parameters and 1e-4 for the rest parameters. For finetuning on Kinetics-Sounds, we train for 24K iterations with a batch size of 32, with learning rate of 0.005 for the classifier and CNN parameters, 1e-4 for the orthogonal parameters and 1e-3 for the rest parameters.
sections.heading: B EXTRA RESULTS FROM THE ABLATION STUDY
sections.heading: B.1 NEGATIVE SAMPLING STRATEGIES
sections.text: We proposed the content-aware negative sampling strategy (CANS) using pairwise l2 distances between CNN embeddings. We introduced two variants of CANS: CANS-Dissimilar that favors negatives that are dissimilar to the positive instance and CANS-Similar that favors negatives that are similar to the positive instance. We chose to use CANS-Similar based on the results from our ablation study presented in the main paper, Table 1 (b).
  Figure 3 (a) in this appendix provides additional evidence that supports our decision. We see that the loss of CANS-Disimilar initially drops rapidly but starts increasing around iteration 7K and continues to increase until around 15K; this is mainly caused by the visual MEP loss shown in Figure 3 (a-2). One explanation for this might that CANS-Disimilar is easier to solve than CANS-Similar, which causes the loss landscape of CANS-Disimilar to contain too many shallow local minima compared to that of CANS-Similar. Recall that we use a learning rate warm-up for the first 6% of iterations during pretraining; this roughly equals to the first 13K (out of 220K) iterations. Given this, we speculate that the model got out of a local minima around iteration 7K (most likely due to the increasing learning rate), and then eventually settled in another (bad) local minima after the warm-up period ended. Compared to this, we observe much milder learning dynamics with CANS-Similar: the loss decreases relatively slowly but steadily, and eventually leaps around 35K to go below the loss of CANS-Disimilar. We, again, believe that this is because CANS-Similar is more difficult to solve than CANS-Disimilar (as shown by the slower decrease in loss values), which caused the resulting loss landscape to contain steeper local minima. Our model eventually found one of those after round 40K of iterations, resulting in a better performing model in the downstream tasks shown in Table 1 (b) of the main paper (the loss kept slowly decreasing after iteration 50K).
sections.heading: B.2 PARAMETER SHARING SCHEMES
sections.text: We proposed a cross-Transformer weight sharing technique, which decomposes weight matrices with low-rank approximation. Recall that each layer of a Transformer contains the multi-head attention layer weights {W q,W k,W v,W b} and the feed-forward layer weights {W c,W d}. We chose to share all six weight matrices across Transformers, though we could have shared any combination of them. To justify this design choice, we empirically compared three variants: (i) Multi-6 that do not share parameters across Transformers, (ii) Multi-6-Part_Att that shares only {W q,W k,W v,W b} (but not {W c,W d}) and (iii) Multi-6-Part that shares all six weight matrices. Figure 3 (b) shows that there is not much difference between all the variants in terms of the loss curves; we chose to use Multi-6-Part that requires the least number of parameters. We showed that our approach outperforms Multi-6 in the ablation study (Table 1 (c-left) in the main paper).
sections.heading: B.3 JUSTIFICATION FOR THE MEP LOSS FORMULATION
sections.text: Since the multimodal Transformer hAV has access to both visual and audio inputs, one might think that the model could “leak” information about visual input into za and information about audio input into zv, which could make MEP trivial to solve. Here we show that this is not the case. By construction, we mask the same positions in audio and visual streams when designing the MEP task, so the model has no access to the masked input even in a cross-modal manner. Empirically, removing the third term in Eq. 6 (LNCE([xa;xv], z̃)) leads to performance degradation in Kinetics-Sounds, i.e., top-1 accuracy 66.7% vs. ours 67.5% (see Table 1), which suggests that solving the MEP task in the multimodal Transformer is beneficial to our model.
sections.heading: B.4 JUSTIFICATION FOR THE CPP LOSS FORMULATION
sections.text: Recall that our CPP loss has two terms; the first term uses the summary embeddings sg and the second term uses output embeddings sh sampled at random positions; see Eq. 7. One could argue that the two terms are redundant as bidirectional Transformers have “one-step” access to all the input embeddings, and thus solving CPP only with the summary embeddings (the first term) would be enough. This is not the case. We encode sh with position-specific information through the time
  embeddings pt, which makes every sh different compared to sg . Empirically, we find that removing the second term of Eqn. 7 (sh) in our CPP loss leads to an inferior accuracy 66.9% vs. ours 67.5% on Kinetics-Sounds, suggesting its importance in learning.
sections.heading: B.5 USE OF MODALITY EMBEDDINGS IN THE MULTIMODAL TRANSFORMER
sections.text: We use modality embeddings mv and ma as part of input to the multimodal Transformer in order to distinguish embeddings coming from visual and audio Transformers. They are learnable weights trained end-to-end with other parameters. Conceptually, incorporating modality-discriminative embeddings is crucial because of our aggressive weight sharing scheme. Without them, the multimodal Transformer will see the output from audio/visual Transformers (ya and yv) as if they are coming from the same distribution because the two Transformers share a large part of weights. Using modality embeddings encourages our model to preserve modality-specific information in the final output, and this empirically leads to performance improvements: ours 67.5% vs. without modality embeddings 67.1% on Kinetics-Sounds.
sections.heading: B.6 ON THE IMPORTANCE OF END-TO-END PRETRAINING
sections.text: Previous work in multimodal visual-and-language tasks (Tan & Bansal, 2019; Lu et al., 2019) point out that using partially fixed Transformers of different modalities is detrimental to multimodal representation learning (c.f., Sun et al. (2019a;b)). We make the same observation in our audiovisual learning scenario. We compare two variants of Multi-6 Part in Table 1 (c), each of which pretrains only the audio (or visual) CNN/Transformer in the first half of pretraining stage and then continues pretraining the remaining weights while fixing the weights of the audio (or visual) CNN/Transformer in the second half. This leads to inferior performance (audio-fixed 62.8% and visual-fixed 63.1% vs. ours 67.5%), which is consistent with the results reported in Tan & Bansal (2019); Lu et al. (2019).