summaries: This paper proposes the use of user comments in addition to videos and titles to learn better representations for retrieval. Since user comments may be loosely related to the video, they use an attention-based mechanism to ignore irrelevant user comments. Their experiments show that using comments using this mechanism leads to better contextualized representations, which lead to competitive results on standard benchmarks.

summaries: The paper looks into video-text retrieval problem.  The claim is that existing works mostly rely on titles and captions.  The paper argues for using user comments.  The challenge is that not all user comments are meaningful or relevant.  Therefore, the paper looks into attention mechanism to filter out the irrelevant content. The main contribution is a context adapter module based on transformer that allows relating visual input and textual input. The base architecture seems to have been derived from CLIP.

summaries: In this paper, the authors proposed a multi-modal video representation learning and retrieval method based on it, taking advantage of visual and text embeddings, particularly users' comments on video sharing platforms. For this, the authors present a new component called Context Adapter Module (CAM), applying skip connections to selectively reflect user comments in the model. According to their experiments, the authors claim that the proposed method outperforms baseline models, by providing more contextualized representations of videos.

summaries: The authors propose a text-based video retrieval method based on context from weakly related user comments. Evaluation shows that considering user comments improves the retrieval performance. Compared to conventional methods, the proposed method shows similar performance with CLIP4Cilip. However, CLIP4clip needs a much larger-scale training, so the superiority of the proposed method is claimed.

abstractText: Learning strong representations for multi-modal retrieval is an important problem for many applications, such as recommendation and search. Current benchmarks and even datasets are often manually constructed and consist of mostly clean samples where all modalities are well-correlated with the content. Thus, current video-text retrieval literature largely focuses on video titles or audio transcripts, while ignoring user comments, since users often tend to discuss topics only vaguely related to the video. In this paper we present a novel method that learns meaningful representations from videos, titles and comments, which are abundant on the internet. Due to the nature of user comments, we introduce an attention-based mechanism that allows the model to disregard text with irrelevant content. In our experiments, we demonstrate that, by using comments, our method is able to learn better, more contextualised, representations, while also achieving competitive results on standard video-text retrieval benchmarks.

authors:
Humam Alwassel
Bruno Korbar
Dhruv Mahajan
Lorenzo Torresani
Bernard Ghanem
Du Tran

title: Selfsupervised learning by cross-modal audio-video clustering

authors:
Yuki M. Asano
Mandela Patrick
Christian Rupprecht
Andrea Vedaldi

title: Labelling unlabelled videos from scratch with multi-modal self-supervision

venue: In NeurIPS,

authors:
Max Bain
Arsha Nagrani
Gül Varol
Andrew Zisserman

title: Frozen in time: A joint video and image encoder for end-to-end retrieval

venue: arXiv preprint arXiv:2104.00650,

authors:
Michael Barthel
Galen Stocking
Jess Holcomb
Amy Mitchell

title: Reddit news users more likely to be male, young and digital in their news

authors:
Emily M. Bender
Timnit Gebru
Angelina McMillan-Major
Shmargaret Shmitchell

title: On the dangers of stochastic parrots: Can language models be too big? In Conference on Fairness, Accountability, and Transparency (FAccT ’21)

authors:
Gedas Bertasius
Heng Wang
Lorenzo Torresani

title: Is space-time attention all you need for video understanding

venue: arXiv preprint arXiv:2102.05095,

authors:
Joao Carreira
Eric Noland
Chloe Hillier
Andrew Zisserman

title: A short note on the kinetics-700 human action dataset

authors:
David Chen
William B Dolan

title: Collecting highly parallel data for paraphrase evaluation

venue: In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies,

authors:
Honglie Chen
Weidi Xie
Andrea Vedaldi
Andrew Zisserman

title: Vggsound: A large-scale audio-visual dataset

venue: IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

authors:
Mark Chen
Alec Radford
Rewon Child
Jeff Wu
Heewoo Jun

title: Generative pretraining from pixels

venue: In ICML,

authors:
Yen-Chun Chen
Linjie Li
Licheng Yu
Ahmed El Kholy
Faisal Ahmed
Zhe Gan
Yu Cheng
Jingjing Liu

title: Uniter: Learning universal image-text representations

authors:
Kevin Clark
Minh-Thang Luong
Quoc V. Le
Christopher D. Manning

title: ELECTRA: Pre-training text encoders as discriminators rather than generators

authors:
Karan Desai
Justin Johnson

title: Virtex: Learning visual representations from textual annotations

venue: arXiv preprint arXiv:2006.06666,

authors:
Jacob Devlin
Ming-Wei Chang
Kenton Lee
Kristina Toutanova. Bert

title: Pre-training of deep bidirectional transformers for language understanding

venue: In ACL,

authors:
Victor Escorcia
Juan Carlos Niebles

title: Activitynet: A large-scale video benchmark for human activity understanding

venue: In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,

authors:
Alon Halevy
Cristian Canton Ferrer
Hao Ma
Umut Ozertem
Patrick Pantel
Marzieh Saeidi
Fabrizio Silvestri
Ves Stoyanov

title: Preserving integrity in online social networks

authors:
Po-Yao Huang
Mandela Patrick
Junjie Hu
Graham Neubig
Florian Metze
Alexander Hauptmann

title: Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models

venue: In Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL),

authors:
Jeff Johnson
Matthijs Douze
Hervé Jégou

title: Billion-scale similarity search with gpus

venue: arXiv preprint arXiv:1702.08734,

authors:
Will Kay
Joao Carreira
Karen Simonyan
Brian Zhang
Chloe Hillier
Sudheendra Vijayanarasimhan
Fabio Viola
Tim Green
Trevor Back
Paul Natsev

title: The kinetics human action video dataset

venue: arXiv preprint arXiv:1705.06950,

authors:
Diederik P Kingma
Jimmy Ba

title: Adam: A method for stochastic optimization

authors:
Mike Lewis
Marjan Ghazvininejad
Gargi Ghosh
Armen Aghajanyan
Sida Wang
Luke Zettlemoyer

title: Pre-training via paraphrasing

venue: arXiv preprint arXiv:2006.15020,

authors:
Mike Lewis
Yinhan Liu
Naman Goyal
Marjan Ghazvininejad
Abdelrahman Mohamed
Omer Levy
Ves Stoyanov
Luke Zettlemoyer

title: Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

authors:
Gen Li
Nan Duan
Yuejian Fang
Ming Gong
Daxin Jiang
Ming Zhou

title: Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training

venue: In AAAI,

authors:
Liunian Harold Li
Mark Yatskar
Da Yin
Cho-Jui Hsieh
Kai-Wei Chang

title: Visualbert: A simple and performant baseline for vision and language

authors:
Yang Liu
Samuel Albanie
Arsha Nagrani
Andrew Zisserman

title: Use what you have: Video retrieval using representations from collaborative experts

authors:
Jiasen Lu
Dhruv Batra
Devi Parikh
Stefan Lee

title: Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

venue: In NeurIps,

authors:
Huaishao Luo
Lei Ji
Botian Shi
Haoyang Huang
Nan Duan
Tianrui Li
Jason Li
Taroon Bharti
Ming Zhou

title: UniVL: A unified video and language pre-training model for multimodal understanding and generation

venue: arXiv preprint arXiv:2002.06353,

authors:
Huaishao Luo
Lei Ji
Ming Zhong
Yang Chen
Wen Lei
Nan Duan
Tianrui Li

title: Clip4clip: An empirical study of clip for end to end video clip retrieval

venue: arXiv preprint arXiv:2104.08860,

authors:
Shuming Ma
Lei Cui
Damai Dai
Furu Wei
Xu Sun

title: Livebot: Generating live video comments based on visual and textual contexts

authors:
Antoine Miech
Dimitri Zhukov
Jean-Baptiste Alayrac
Makarand Tapaswi
Ivan Laptev
Josef Sivic

title: Howto100m: Learning a text-video embedding by watching hundred million narrated video clips

authors:
Pedro Morgado
Nuno Vasconcelos
Ishan Misra

title: Audio-visual instance discrimination with cross-modal agreement

venue: arXiv preprint arXiv:2004.12943,

authors:
Mandela Patrick
Yuki M Asano
Polina Kuznetsova
Ruth Fong
Joao F Henriques
Geoffrey Zweig
Andrea Vedaldi

title: Multi-modal self-supervision from generalized data transformations

venue: arXiv preprint arXiv:2003.04298,

authors:
Mandela Patrick
Po-Yao Huang
Yuki Asano
Florian Metze
Alexander Hauptmann
João Henriques
Andrea Vedaldi

title: Support-set bottlenecks for video-text representation learning

venue: arXiv preprint arXiv:2010.02824,

authors:
Mandela Patrick
Yuki M. Asano
Polina Kuznetsova
Ruth Fong
João F. Henriques
Geoffrey Zweig
Andrea Vedaldi

title: Multi-modal self-supervision from generalized data transformations, 2021

authors:
Jesús Andrés Portillo-Quintero
José Carlos Ortiz-Bayliss
Hugo Terashima-Marín

title: A straightforward framework for video retrieval using clip

venue: arXiv preprint arXiv:2102.12443,

authors:
Alec Radford
Jeffrey Wu
Rewon Child
David Luan
Dario Amodei
Ilya Sutskever

title: Language models are unsupervised multitask learners

venue: OpenAI Blog,

authors:
Alec Radford
Jong Wook Kim
Chris Hallacy
Aditya Ramesh
Gabriel Goh
Sandhini Agarwal
Girish Sastry
Amanda Askell
Pamela Mishkin
Jack Clark

title: Learning transferable visual models from natural language supervision

venue: arXiv preprint arXiv:2103.00020,

authors:
Colin Raffel
Noam Shazeer
Adam Roberts
Katherine Lee
Sharan Narang
Michael Matena
Yanqi Zhou
Wei Li
Peter J Liu

title: Exploring the limits of transfer learning with a unified text-to-text transformer

venue: arXiv preprint arXiv:1910.10683,

authors:
Sylvestre-Alvise Rebuffi
Hakan Bilen
Andrea Vedaldi

title: Learning multiple visual domains with residual adapters

authors:
Mert Bulent Sariyildiz
Julien Perez
Diane Larlus

title: Learning visual representations with caption annotations

venue: In ECCV,

authors:
Weijie Su
Xizhou Zhu
Yue Cao
Bin Li
Lewei Lu
Furu Wei
Jifeng Dai

title: VL-BERT: Pre-training of generic visual-linguistic representations

authors:
Chen Sun
Fabien Baradel
Kevin Murphy
Cordelia Schmid

title: Learning video representations using contrastive bidirectional transformer

venue: arXiv preprint arXiv:1906.05743,

authors:
Chen Sun
Austin Myers
Carl Vondrick
Kevin Murphy
Cordelia Schmid

title: Videobert: A joint model for video and language representation learning

authors:
Hao Tan
Mohit Bansal

title: Lxmert: Learning cross-modality encoder representations from transformers

venue: In EMNLP,

authors:
Ashish Vaswani
Noam Shazeer
Niki Parmar
Jakob Uszkoreit
Llion Jones
Aidan N. Gomez
Lukasz Kaiser
Illia Polosukhin

title: Attention is all you need

authors:
Subhashini Venugopalan
Marcus Rohrbach
Jeff Donahue
Raymond Mooney
Trevor Darrell
Kate Saenko

title: Sequence to sequence – video to text

venue: In Proceedings of the IEEE International Conference on Computer Vision (ICCV),

authors:
Hu Xu
Gargi Ghosh
Po-Yao Huang
Dmytro Okhonko
Armen Aghajanyan

title: Videoclip: Contrastive pre-training for zero-shot video-text understanding

venue: arXiv preprint arXiv:2109.14084,

authors:
Jun Xu
Tao Mei
Ting Yao
Yong Rui

title: MSR-VTT: A large video description dataset for bridging video and language

venue: In CVPR,

title: Msr-vtt: A large video description dataset for bridging video and language

venue: In Proceedings of the IEEE conference on computer vision and pattern recognition,

authors:
2022 Youngjae Yu
Jongseok Kim
Gunhee Kim

title: A joint sequence fusion model for video question answering and retrieval

authors:
MSVD MSVD Chen
Dolan

heading: 1 INTRODUCTION

text:
Training large scale multi-modal models from paired visual/text data from the web has seen great success in video understanding and retrieval. However, typically only the caption or “alt text” is used during training, ignoring potentially relevant text present on the web page, such as user comments, tags, descriptions and other metadata.
The challenge in using this extra context is that it may often be not directly relevant to the video (a comment “cool video!”), or it may be relevant but non-distinctive (a tag “cat” could apply to many videos). Similarly, other modalities can also exhibit this behavior. For example, current work that learns from audio-visual correspondence (Asano et al., 2020; Alwassel et al., 2020; Morgado et al., 2020) need clean datasets such as VGGSound (Chen et al., 2020a) to learn meaningful correspondence between videos and sound, whereas online videos tend to have for example background music that replaces the actual sounds happening in the video.
In this paper, we propose a method that can take advantage of this auxiliary context while simultaneously filtering it for meaningful information. Most current models enforce a strict correlation between the different input modalities under the assumption that all are informative of the content. The main intuition of our work is that when training a model on partially unrelated data, we need to introduce a mechanism that allows the model to discount auxiliary data when it is not helpful for the task.
To this end, we build a model with a hierarchical attention structure. Current representation learning models that are based on transformer architectures already exploit the idea of an attention mechanism to model the correlation between different parts of an input signal. For example in text understanding, the attention mechanism is applied per word, allowing the model to understand the structure of natural language. Even though in principle one could use the same scheme to model the importance of different text inputs on a per-word basis, we find that this makes it difficult to learn the individual importance of inputs. Moreover, due to the computational complexity of current transformers (squared with sequence length) this approach would only work for a small number of short comments. We thus add a second layer of attention per processed input that allows the model to assess the amount of information at a higher level of features. We find that this mechanism aligns well with the intuition that some inputs are very relevant to the problem and others can be disregarded.
In our experiments we show that we can indeed learn meaningful information from user comments that translates to good zero-shot retrieval performance on common benchmarks. Additionally, we can show that the model can correctly identify whether auxiliary information is informative of the content of a video or not.
The ability to incorporate auxiliary contextual information also opens up possibilities for useful applications. In the video retrieval setting our method can be used to iteratively refine a text descriptor with new inputs as shown in Fig. 1, allowing incremental searching. In the zero-shot video classification setting (i.e., “retrieving” the correct class description prompt) the prediction for an ambiguous video can be steered towards the correct class using surrounding text from a webpage or user hints (Fig. 4).

heading: 2 RELATED WORK

text:
In this work, we focus on multi-modal learning with a particular focus on learning video-text encoders for retrieval by proposing a novel, multi-modal adaptation module.
Video-text pretraining. Originating from the NLP domain, where the transformer architectures has been a key ingredient and subject to optimization in a multitude of ways (Vaswani et al., 2017; Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2019; Lewis et al., 2020b; Clark et al., 2020; Lewis et al., 2020a), it has recently found applications in the vision-language domain. For example, recent works have leveraged transformers to learn generalizeable image (Desai & Johnson, 2020; Sariyildiz et al., 2020; Chen et al., 2020b) or multi-modal image-text (Li et al., 2019; Lu et al., 2019; Tan & Bansal, 2019; Su et al., 2019; Li et al., 2020; Chen et al., 2019) and video-multilingual text (Huang et al., 2021) representations. A few works (Sun et al., 2019b;a; Zhu & Yang, 2020; Luo et al., 2020) combine visual and text modalities as inputs to a BERT model to simultaneously learn semantic video and text representations. For representation learning, the availability of large-scale datasets such as HowTo100M (Miech et al., 2019) has enabled more effective pretraining of video-text representations for multiple downstream tasks. More recently, Patrick et al. (2020b) show that adding a generative objective to contrastive pretraining can yield gains in video-text downstream tasks. Based on the CLIP model (Radford et al., 2021), which works well even without finetuning for some retrieval tasks (Luo et al., 2021), Bain et al. (2021) train video-text CLIP-initialized models by gradually scaling up video training from image training and a custom dataset. While we also start with a CLIP initialization as in Luo et al. (2021), the focus of our paper lies in developing a novel method for leveraging user comments, a modality that has previously been overlooked in the text-video retrieval literature, as a valuable source of information. As is standard, the pretrained representations are subsequently evaluated on smaller datasets such as MSVD (Venugopalan et al., 2015) and MSR-VTT (Xu et al., 2016a).
Multi-modal domain adaptation. While residual adapters for domain adaptation have been explored for uni-modal models such as CNNs, e.g. in Rebuffi et al. (2017), there are no works that translate
this concept to the multi-modal domain, where cross-modal learning dominates (Alwassel et al., 2020; Asano et al., 2020; Morgado et al., 2020).
Multi-modal pretraining. There is little prior work using user comments as additional context in multimodal settings. The use of user comments and reactions to refine predictions has been discussed by Halevy et al. (2020) for minimising harms on facebook.com. Overall, we are the first to demonstrate that user comments can be used as a complementary modality when learning video-text representations.

heading: 3 METHODS

text: In this section we will first recap the mechanism behind current contrastive, multi-modal representation learning methods that rely on clean data. We will then introduce our Context Adapter Module that allows learning from the auxiliary modality through an attention mechanism. Finally, we will describe how we can extend an existing backbone for images to videos, to be able to leverage large, pretrained models.

heading: 3.1 BACKGROUND

text:
In multi-modal representation learning we are given a datasetX ofN samples xi ∈ X , i ∈ {1, . . . N} that individually consists of different signals. Most previous work focuses on two modalities and we will—for now—also adhere to this standard to simplify the notation. This means that each input sample xi = (vi, ti) is a pair of—in our case—a visual input vi ∈ V and its associated text, often the title, ti ∈ T , 1 ≤ i ≤ N . The goal is now to learn mappings fv : V 7→ Y, fv(vi) = φv,i and ft : T 7→ Y, ft(ti) = φt,i from each of the modalities to a d-dimensional, joint embedding space Y = Rd. Recent methods, such as Radford et al. (2021), learn the mapping (in their case from images and their captions) to the embedding space with a double contrastive loss over a mini-batch B ⊂ X using an affinity matrix A computed between all pairs of samples in the batch:
Aij = 〈 φv,i√ τ‖φv,i‖ , φt,j√ τ‖φt,j‖ 〉 (1)
An entry Aij measures the similarity between the embeddings φv(vi) and φt(ti) via cosine similarity that is scaled by a temperature parameter τ . The idea is now to maximize the similarity between the embeddings from the same sample, i.e. the diagonal of A and minimize all non-diagonal entries. This can be achieved efficiently using a double-contrastive formulation that operates across columns and rows of A,
L(A) = 1 2 |B|∑ i=1
Aii log ∑|B| j=1 expAij +
Aii log ∑|B| j=1 expAji . (2)
This formulation has the neat effect that it accomplishes maximizing the diagonal entries and minimizing all other entries of A in one self-balancing formulation. However, it makes the critical assumption that both modalities are equally informative of each other. In the case of sometimes irrelevant data, or when one modality has much less information content than the other (e.g. “nice video!”), this assumption does not hold and training with this objective will result in a very volatile learning objective and thus a sub-optimal joint embedding.
In the next section we will introduce our Context Adapter Module that is able to deal with this type of inputs by allowing it to discount information when it is not relevant for the context.

heading: 3.2 CONTEXT ADAPTER MODULE

text:
In order to capture and filter the relevant information from the comments, we propose a transformerbased Context Adapter Module (CAM) which operates in a residual fashion, additively adapting either the visual or text branch of CLIP with contextual information obtained from the comments (see Fig. 2). Formally, we are now adding another modality—the comments—to the input which
extends it to xi = (vi, ti, ci,1, . . . , ci,M ) with ci,k ∈ T . To reduce clutter in the notation, we have defined a fixed number of comments M for each sample. Since both, title and comments, share the same underlying modality, namely text, we can leverage the same encoder to transform comments to embeddings ft(cik) = φc,ik.
As we expect the comments to be sometimes unrelated, our Context Adapter Module needs a mechanism to discount off-topic comments and update the primary modality φv(vi) or φt(ti), steering it in the most informative direction.
We introduce this mechanism as a function of both the primary modality and the comment embeddings φc,ik, as we want to compare the informativeness of all these inputs at a high level. To this end, we design adapter modules gv and gt that extract information from the comments in the form of a residual:
φ̂oi = φoi + go(φoi, φc,i,1, . . . , φc,i,M ) , o ∈ {v, t} (3)
With the adapted embeddings φ̂vi and φ̂ti we recompute the affinity matrix (now Â) (Eq. 1) and use it for the loss L(Â). This design has several advantages. On one hand, extracting “only” a residual from the auxiliary inputs cik means that the model is easily able to ignore them by predicting g(·) = 0. On the other hand, this effectively allows us to skip the adapter module when we benchmark the model on a dataset that does not have comments, while still learning the joint embedding from richer data.
In practice, we implement g as a small transformer architecture. Rather than operating on tokenised words, this transformer operates on embeddings (φvi and φti) themselves, taking as input the encoded feature from the branch to be adapted, along with comment features φc,ik. By treating embeddings as tokens in their own right, we allow the embeddings to attend to each other and learn what combinations of the inputs should be used to update the original feature through the residual connection.
Additionally, to avoid bleeding information between the two modalities through the Context Adapter Module, during training, we only adapt either the video embedding with gv or the text embedding with gt. If we would use both adapters simultaneously, there is a trivial solution that minimizes the loss L: when the adapters learn to remove the original embedding through the residual, e.g. go(φoi, {φc,i,k, }) = −φoi + φc,i,1 both adapted embeddings become the same φ̂vi = φ̂ti which trivially maximizes their similarity, thus preventing the model to learn a meaningful modality alignment. To prevent the model from learning a transformation of the embedding space through the residual, we train only one adapter at a time. We also consider the case where gv = gt and choose a random modality to adapt for each minibatch, leading to an adapter that is agnostic to the source of the primary embedding.

heading: 3.3 FROM IMAGES TO VIDEOS

text:
To leverage the capacity of large pre-trained computer vision models, we adopt the architecture by Radford et al. (2021) as our backbone models fv and ft. While this transformer was trained on a huge volume of image-text data, it cannot be applied directly to videos since it is built for images and has no temporal extent. To take advantage of the temporal information present in video
data, we use the Divided Space-Time attention mechanism recently introduced in the TimeSformer architecture (Bertasius et al., 2021). We modify the image transformer architecture by adding patchwise self-attention across 8 frames in time to each of the 12 residual attention blocks, followed in each case by a zero-initialised linear layer. We also add a learned temporal position embedding which is summed to the input and again zero-initialised. The initialisation is transparent, such that when loading pretrained weights trained from images, at initialization time, the modifications do not affect the inference of the model. During training, the model can then gradually activate the additional temporal components to learn from the temporal information of a video. Full details on the architecture are provided in the appendix.

heading: 4 EXPERIMENTS

heading: 4.1 IMPLEMENTATION DETAILS

text:
We use CLIP (Radford et al., 2021) as the initialisation for the backbone. Our concrete implementation of the CAM g is a 2-layer transformer, consisting of two residual multihead self-attention blocks. The input consists of M + 1 input embeddings (for the M comments and title/video embedding φoi) having 512 dimensions each. Each block performs 8-head self-attention on the inputs, followed by two linear layers with output size 2048 and 512 respectively. LayerNorm normalisation is used, along with GELU activation following the first linear layer. From the M + 1 outputs of the transformer, we take only the output at index 1 (the index of the text/video embedding), and pass it through a final 512× 512 linear layer. We use learning rate 0.001 for training the CAM, and 1× 10−6 when training the entire model, using the Adam Kingma & Ba (2015) optimizer.
All implementation and architecture details can be found in the appendix.

heading: 4.2 PRE-TRAINING DATASETS

text:
RedditVC. We collect a dataset “RedditVC” of videos along with their titles and comment threads from social news site reddit.com, using their provided API. The videos are collected and used in a manner compatible with national regulations on usage of data for research. Unlike most curated video datasets, this data is more representative of the types of videos shared “in the wild”, containing a large proportion of videogames, screenshots and memes.
Using a classifier trained on a small amount of labelled data, we estimate that videogame footage makes up 25% of examples, other screenshots, memes and comics make up 24%, live action footage is 49% and artistic styled content (such as drawn animation) is 2%. The average video length is 33s.
From 1 million raw videos collected, we perform deduplication and filtering, ending up with a training set of 461k videos, and validation and test sets of 66k videos each. For the video evaluation results in Table 1, we use a subset of the test set, consisting of 5000 videos with at least three comments each.
KineticsComments. As an additional video dataset with comments, we construct a dataset based on Kinetics-700 (Kay et al., 2017; Carreira et al., 2019), for which we download the videos along with associated YouTube metadata including title, description and comments. We translate non-English titles and descriptions into English using a commercial translation API. We use the title as the primary text modality, and for auxiliary context we use comments and (when training) sentences from the description. This leaves us with 484,914 videos for training, each being around 10s. We also construct a test set, consisting of videos from the Kinetics test set for which we have at least 3 comments, giving a set of 2700 videos which we use to evaluate our method in Table 1.

heading: 4.3 EVALUATING THE CONTEXT ADAPTER MODULE

text:
We evaluate our Context Adapter Module on the above described datasets with comments in Table 1. We find that on both datasets adding comments boosts the retrieval performance significantly, confirming the value of the modality. Further, moving from images to videos also improves the performance. Since KineticsComments (human actions) is quite different to RedditVC (broad range of videos, games, etc.) combining both datasets bridges the domain gap and improves the evaluation performance on KineticsComments.
Baseline Comparisons. Since the main purpose of the Context Adapter Module is to extract meaningful information from sometimes unrelated auxiliary data, we perform most of the experiments in this section with an image/text backbone using a single video frame instead of a video/text backbone to reduce the computational burden of the evaluation.
In Table 2 we evaluate the efficacy of the proposed Context Adapter Module against various baselines in various settings on our RedditVC dataset. The most basic baseline is achieved by not using any form of context adaption, which degenerates the model to the one proposed by Radford et al. (2021).
Across all experimental settings, we find that finetuning the backbone architecture helps to improve the performance, while using the proposed CAM with frozen backbones (rows h,k) even outperforms the naive baseline of row (a). We also compare to two baselines. The first baseline is randomly replacing the title features with comments during training (Table 2(c-d)), however, this does not yield any improvement over no adaptation. In the second baseline we average the title features φt and the
comment features φc which can be seen as a very simple adapter module (Table 2(e-f)). This yields a small benefit when both training and evaluation use averaging, but falls behind utilizing proposed CAM on the visual modality (Table 2(l)). Surprisingly, we find that training with comments performs best when they are added to the image branch (rows j-l), as opposed to the text-branch (rows g-i). We hypothesize that this is because the comments can better adapt the semantically richer feature of the visual modality as opposed to the text, which can often be as short as a single word.
Adapting Different Modalities. Since both the visual embedding φv and the text embedding φt, come from the same embedding space and should be similar for the same sample, we perform an experiment where we swap the embeddings during evaluation time. In Table 3 we find that there are still differences between φv and φt and swapping results in decreased performance. When we train both adapters we can achieve good performance in both cases.
Auxiliary Information. Additionally, Table 4 shows how varying the source of the text at train and test time impacts the retrieval results. Whilst the performance of the model trained with image labels degrades when given comments at evaluation time, a model trained with comments still benefits from image labels at test time. As expected, the performance of both models is negatively impacted when tested with uncorrelated random words as additional text. However, the model trained with image labels does noticeably worse in this setting. This suggests that it has become overly reliant on image labels and easily fooled by irrelevant information, which overrides the title feature.
Varying the number of comments. In Fig. 3 we vary the number of comments during training and evaluation time. Training on more comments results in better retrieval results when evaluated on the same number of comments. However, while there is little difference for models trained with only a few comments, we found that increasing the number of comments at train time leads to models that are more robust to different number of comments at test time, indicating that training with more
comments learns an overall better context adapter module that is then able to work efficiently in various settings.

heading: 4.4 COMPARISON TO THE STATE OF THE ART

text:
In order to compare to other methods, we use the text-to-video retrieval benchmarks MSR-VTT (Xu et al., 2016a) and MSVD (Venugopalan et al., 2015). While these benchmarks do not directly take into account the novel ability of our method to consider contextual information, they give a measure of the transferability of the learned text and video features. For evaluation, we skip the CAM and use φvi and φti directly since these benchmarks do not involve comments. Fine-tuning a learned representation only yields an indirect measure of the original model’s quality. Since we are interested in the immediate quality of the learned representations, we focus this evaluation on zero-shot performance on the datasets, without fine-tuning. Nonetheless, fine-tuning results can be found in the appendix.
Datasets. For MSVD (Table 5), we use the test split consisting of 670 videos and approximately 40 captions each.
For MSR-VTT (Table 6), we use the 1k-A or “jsfusion” evaluation split of the data, consisting of 1000 (video,text) pairs and 20 captions per video.
Moreover, we show the zero-shot performance of our method on 2 additional datasets. In Table 7 we report the Text-to-Video retrieval results on the ActivityNet (Fabian Caba Heilbron & Niebles, 2015) dataset. We evaluate on the “val 1” split, as in SSB (Patrick et al., 2020b) who we compare to. In Table 8, we show our Text-to-Video retrieval performance on the LSMDC Rohrbach et al. (2017) dataset, where we use the test set comprising of 1000 videos.
Finally, we report the Video-to-Text retrieval results for MSVD and MSR-VTT, which follow similar patterns, in the Appendix.
Results. Across all benchmarks we improve over all other methods in zero-shot retrieval performance. On MSR-VTT our performance is even slightly better than CLIP4Clip (Luo et al., 2021), that uses the same backbone initialization as us but additionally trains on HowTo100M which is more than 100 times larger than our dataset.
It is interesting to note, that we are the only method that evaluates zero-shot performance on such a wide spectrum of datasets which shows good generalization performance of the learned representations.
Using comments, videos and more data. In Tables 6, 9 and 10 we also show the performance of our model in different variants.
Across all three benchmarks, our model trained with comments is consistently better than the variant trained without comments. This means that even though the evaluation is performed without comments (none of these datasets provides user comments), the additional context during training has improved the underlying video-text representations.
Similarly, using video instead of a single image improves the results as well. The main difference here, however, is that the video is also available during evaluation. Thus, the observation is that video provides more or a better context for retrieval than single images.
Finally, we find that adding Kinetics to the training set has only a small effect on the quality of the learned representation and can—depending on the domain gap between Kinetics and the benchmark dataset—even affect the results slightly negatively.
Thus, while achieving state-of-the-art performance, we can also show that comments provide a strong signal to improve multi-modal representation learning, even when comments are not available during test time.
We find that the context adapter can be led to override the information in a title if we adversarially craft comments that all point to different content. Qualitative examples of this can be seen in Fig. 4. The model without comments, correctly associates the image with a cookie (jar), however when adding a comment about a “dog” the model prefers the dog label over the cookie label. More examples can be found in the appendix (Appendix A.5 and the results browser in the supplementary material).

heading: 5 CONCLUSION

text:
We have presented the Context Adapter Module, which is able to extract information from auxiliary input sources for learning a joint, multimodal embedding. In our experiments, we are able to show that the model improves the rep-
resentation quality of video-text embedding learning when adapting the representation with user comments. Moreover, it is able to identify whether an auxiliary input is relevant to the content in the other modalities or not. This mechanism could, for example, be used to filter datasets for meaningful auxiliary content. The model can be initialized from existing, large-scale image-text backbones and the context adapter learns from a comparatively small number of examples (<500k). The module itself is modality-agnostic and can potentially be used in a large variety of multi-modality representation learning tasks such as audio-visual learning, or to combine the outputs of unreliable expert models.

heading: REPRODUCIBILITY AND BROADER IMPACT

text:
Online data collected from user-generated content are subject to all sorts of biases and potentially problematic content. We have carefully selected the discussion topics from which we source the data, to include topics with a lower chance of problematic topics such as pets, art, jokes and games. On top of that we filter text content with Detoxify (Hanu & Unitary team, 2020) for harmful content. The use of a content filtering text model such as Detoxify can introduce its own biases. Although we have used the ‘unbiased’ version of Detoxify to minimise these, there might still be particular words that are associated with offensive or hateful language that will likely be strongly correlated with high toxicity scores, regardless of the intent of the author. This could lead to overfiltering non-toxic text that contains particular identity terms, which might result in discrimination against already vulnerable groups. Moreover, the user-base of Reddit is predominantly young, male and western (Barthel et al., 2016) and thus inherently there will be certain biases present regardless of how thoroughly the dataset is filtered. The use of a primarily English language data source and pretrained model means learned concepts will be far from globally representative. The social biases discussed in Section 7.1 of Radford et al. (2021) all still apply. Additionally, as the data is downloaded from a public website, content creators and users could not be asked for consent to be included in the dataset.
The presented context adapter mechanism has a wide applicability across different multi-modal tasks where one input source is expected to sometimes be uninformative. However, as detailed in the previous section, when trained on online data, the model has no mechanism to distinguish between true and false statements and will simply learn from correlations. It has been repeatedly shown that algorithms trained on online user content are often unfit for any kind of production system and pose risks (Bender et al., 2021). Our work and analysis is for research purposes only and we strongly advise against using models trained on the RedditVC dataset in non-research applications.
Given the above discussion on impact and biases, we refrain for now from publicly releasing the RedditVC dataset or models trained on it, as their intent is for research proposes only and not fit for real-world applications. However, we are planning to release the comments collected for the KineticsComments dataset as the content of the videos is much more constrained, which results in easier filtering of the data.
For further reproducibility of the general approach we will release the code. Additionally, models trained on datasets other than RedditVC will be released as well.

heading: A APPENDIX

heading: A.1 QUALITATIVE EXAMPLES

text:
In Fig. 5 we adapt the text branch, similar to Fig. 1 of the main paper. The example in the second row of Fig. 5 shows how our Context Adapter Module can leverage the comments to learn that the content is indeed about parrots, as opposed to dogs. The fourth row shows that without comments, the title alone can be extremely ambiguous while comments can again guide the model to retrieve relevant videos of drumming.
We provide examples where the video branch has been adapted in Fig. 6. In most cases, the retrieved titles are broadly related to the video thumbnails. However, when provided with the comments, the retrieved titles become more specific to the videos. For example, in the example from the second row of a screenshot from Mario Kart, the retrieved titles are generally about games e.g. The Castle or Sun Haven, whereas when adapting the video with the comments, the model retrieves titles specifically about Mario Kart. Similarly, in the example from the last row, the model seems to get confused about the content of the video when deprived of the comments, which provide the necessary context about feeding a fish.
Finally, in Fig. 7, we show the saliency of comments with regards to a given video and title. For this, we use the approach of masking out each comment in turn, allowing us to visualise the effect of each individual comment on the network output. We compare the output descriptor when including all comments to the descriptors with a comment masked, using the inner product as a score of similarity, and present the comments sorted from lowest to high, the expectation being that an uninformative comment will not cause a large shift in the descriptor (so will still have high similarity when excluded) whereas a salient comment will cause a larger shift (and so a lower similarity when excluded). We show results for adapting both the text branch (left) and visual branch (right), and observe that, as expected, uninformative comments such as “That was great!” and “Possibly?!?!?!?! Lol” cause little change to the descriptor, whereas comments related to objects in the video cause a larger shift. This demonstrates that the method is able to pick out and filter the relevant information.

heading: A.2 TRAINING DETAILS

text: The majority of experiments were conducted on a rented 4xA100-40GB GPU server costing approximately 170USD per day, over the course of three months. Image models (using batch size 128) and video models without comments (using batch size 50) could train on a single 40GB GPU. For TimeSformer models the visual branch was processed on a separate GPU (when training with CAM and batch size 50) or pair of GPUs (for finetuning on video benchmarks with batch size 128). Pretraining the adapter on images takes approximately one hour per epoch. Training the full video model with CAM takes approximately 6 hours per epoch. For the video experiments, we first train the CAM for 5 epochs with the backbone frozen, and then train the rest of the network for one epoch, with the backbone modified to have temporal attention as described in Section 3.3. We use the CAM with 5 comments, and adapt the visual branch of the model.

heading: Method R@1 R@5 R@10 MedR

text:
SSB (Patrick et al., 2020b) 21.4 46.2 57.7 6.0 CLIP (Portillo-Quintero et al., 2021) 59.9 85.2 90.7 1
Ours (no comments) 62.5 87.0 92.8 1 Ours (no TimeSformer) 59.1 85.1 91.0 1 Ours 61.6 87.9 93.4 1 Ours (with Kinetics) 61.3 87.0 92.4 1
We use both photometric and temporal data augmentation. For photometric augmentation we employ random crops (0.5− 1.0 scale), random horizontal flipping, and colour jitter (brightness, contrast, saturation, hue). For temporal augmentation, we first temporally subsample the input frames (which are often 30fps) according to a random stride selected uniformly from (4, 8, 16, 32) and then choose
a random 8-frame segment uniformly. We normalise inputs using the same preprocessing as Clip (ImageNet mean and standard deviation, 224× 224 input size). At evaluation time we use a temporal stride of 16 and split the video into 8-frame chunks, taking the average of the descriptors of the chunks.
Harder the struggle, longer the snuggle.
#YOU CANT ESCAPE THE CUDDLES
“Video, video, always with the video! ...but if you must, behind the ears mom, let’s go.”
He seems very mellow! And he is very handsome.
I’ll check your profile out
Retrieved titles w/o adapting Retrieved titles with adapting
Thank you!
Looks really good anyway. Great work.
Just add another Vader cutaway, could be quite humorous
I'll have another look at it and see if I can make it look good
With: James Earl Jones and David Prowse As Death Vader
Comments
Rip to the guy who got blue shelled last second.
Those green shell hits were clean
Lol yeah, good point. Mk is definitely Chaos.
It is, although it's much easier in 200
Video thumbnail
Appreciate it man the vibes roll on
Sounds fire bro, keep going! Hozier rocks
exactly! Thanks
sounds great! got the birds chirping in the background just like in the recording down pretty well too
Sweet man thanks. Keep at it
today my cat woke up more loving than usual (sorry if I wrote something wrong) Morning routine of kneading and face smushing with my best guy I don't know why, but my cat likes my bed so much. The first thing he does when we let him in is go to my room. He just enjoys it so much. This is his moment of happines, and mine as well. He just looks so happy! Sound on! Luna like to purr on me in the mornings My kitty Mordecai was tucking me in when I didn’t feel well Sound on! Luna like to purr on me in the mornings Woke up with her on me My kitty Mordecai was tucking me in when I didn’t feel well It’s been a crappy week, but his cuddles definitely help sooth my soul. I think she missed me when I was traveling. (Sound on!)
Why did I make this fan edit? Because of Obi-Wan. Made another rotoscope test [oc] My tribute to the Original Trilogy - just over 6,000 pieces. Prequels next - they will eventually all join up. Full video in comments � I wanted to visualise why TROS' ending disappointed me so much, so I edited ROTJ to have an ending that's more similar to certain aspects of TROS'. Feel free to discuss. My first attempt at video editing lol. I call this one The High Ground. Gonna want the audio on this one "The Chosen One" Star Wars 3D Lenticular Fan Artwork ... created by interlacing strips in alternating orders so light reflecting at different angles shows multiple images ... hand drawn & self-produced! Star Wars/Interstellar - My most ambitious and timeconsuming project yet. Enjoy! I wanted to visualise why TROS' ending disappointed me so much, so I edited ROTJ to have an ending that's more similar to certain aspects of TROS'. Feel free to discuss.
My tribute to the Original Trilogy - just over 6,000 pieces. Prequels next - they will eventually all join up. Full video in comments � [OC] Qui-Gon & Obi Wan VS Darth Maul Except Darth Maul Fights With a 20 BLADED LIGHTSABER.
Retrieved titles w/o adapting Retrieved titles with adaptingComments Video thumbnail
Retrieved titles w/o adapting Retrieved titles with adaptingComments Video thumbnail
Retrieved titles w/o adapting Retrieved titles with adaptingComments Video thumbnail
Retrieved titles w/o adapting Retrieved titles with adaptingComments Video thumbnail
It's still Friday in Hawaii! Been 2 months on my first guitar (Guild Om-150CE)... learning from Youtube Sat to work on an old project. - it ended up as something completely different. Fingerpicking + open chords + listening to the Beatles last week. What’s your thoughts? This is my first time posting what I play on guitar, so I was pretty nervous, anyways this peace is called "Etiuda quasi tarantela". My first post on Reddit of my singing. My take on Santeria by Sublime Stumbled across leaves on a vine while modelling on my guitar. Probably wrong and in a different key but thought you guys might enjoy it Yo yo yo this is a song I'm working on called "Lovers and Friends" - do y'all dig it? Sat to work on an old project. - it ended up as something completely different. Fingerpicking + open chords + listening to the Beatles last week. What’s your thoughts? Sadness and Sorrow on mandolin (voice memo) - my first time letting anyone besides close friends and my teacher hear me play My first post on Reddit of my singing. My take on Santeria by Sublime My first post here! The song is called Cigarettes and Rain. I'm pretty new to songwriting so feedback is welcome!
Full tour of my fully decorated medieval island kingdom, 6 months of work How it started/How it's going - Crown Trick. This is one year of progress between the playable demo and the final version of the game. Still can't believe that we are releasing the game tomorrow on Steam and Nintendo Switch If any wonder what if when Marvelous Merchant switch places + New Location Does the ghost in The Castle come with the DLCs or...? One of my personal favorite parts of our game Sun Haven-- The Wishing Well
[MK8DX] Sometimes the Blue Shell can really help out. [MK8DX] Is my copy of Mario Kart defective, or what? I suck at Mario kart double dash but got 2 specials in a row (I'm best at ds) [MKDS] Blue Shells just don't care about walls [MK8DX] That was they funniest Mario Kart moment I've ever had (look at the Koopa)
Lol thanks. Everything that could’ve gone right with the end of this race did. The exact opposite of some races where you go from 1st to last within seconds (here’s looking at you, Moo Moo Meadows). Those green shell hits were clean
We randomly mask out comments with probability 0.5. We randomly skip adding the residual from the adapter with probability 0.5, which ensures that unadapted descriptors are also used in the loss and so the backbone network can still be used without the adapter.
All retrieval experiments are GPU accelerated using the FAISS1 library.
A.3 VIDEO-TO-TEXT RETRIEVAL
We report our zero-shot Video-to-Text Retrieval evaluation results on MSVD and MSR-VTT in Table 9 and Table 10.

heading: A.4 KINETICS COMMENTS

text:
In this section we will describe the details for the additional comments we retrieve for the Kinectics700 dataset. In Table 11 and Fig. 8 we show the distribution of the number of comments in the dataset. We collect a maximum of 10 comments and exclude videos without comments, which leaves
1https://github.com/facebookresearch/faiss
us with 111 920 videos of the originally 650 000 video clips. The majority of videos has one or two comments available. Nonetheless, our experiments show (Table 1) that our method can make use of this information to learn better representations.

heading: A.5 ADDITIONAL FAILURE CASES

text: In Fig. 9 we show additional failure cases. We find that vary vague comments “Why” or generic expressions “Ain’t his fault” can distract the model from the title. In the last example, the model does not capture the concept of a sad dog due to the mention of “happy” in the comments.

heading: A.6 DATASET DETAILS

text:
We report additional details on the benchmark datasets used in the paper.
ActivityNet ActivityNet Fabian Caba Heilbron & Niebles (2015) is a video dataset with 200 classes, 100 videos per class and 1.5 activity instances per video. The dataset contains 648h of video. There are 10,024 training videos (15,410 instances) and 4,926 validation videos (7,654 instances). The dataset itself has no license and is provided as an annotated list of youtube links. The copyright of the individual videos remains with their creators.
MSR-VTT MSR-VTT Xu et al. (2016b) is a video dataset with 10 categories, and 10,00 clips from 7180 videos. For training we use the 6.5k video training set with 20 captions each. It is annotated with 200,000 sentences of video descriptions created by crowd workers. The dataset itself has no license and the copyright of the individual videos remains with their creators.
MSVD MSVD Chen & Dolan (2011) consists of 1200/20/670 videos for training/validation/testing. The dataset itself has no license and the copyright of the individual videos remains with their creators.
LSMDC Large Scale Movie Description Challenge (LSMDC) Rohrbach et al. (2017) consists of 118K video clips from 202 movies. We evaluate on a test set of 1,000 videos from movies disjoint
from the train and val sets, as done in (Bain et al., 2021). The dataset itself has no license and the copyright of the individual videos remains with their creators and the dataset is solely to be used for research purposes.

heading: A.7 DATASET CURATION

text: We use the GPU implementation of the FAISS similarity search toolkit (Johnson et al., 2017) to efficiently deduplicate the dataset by indexing the video thumbnail embeddings obtained from a ResNet18. These indices are then used to discard video entries with a high similarity to other posts.

heading: A.8 CONFIDENCE INTERVALS

text: We report the mean retrieval results in Table 14 and the confidence intervals of the best two image models from the ablation study (Table 1, main paper), namely the models adapting the visual branch, with and without finetuning the backbone. These are computed across five independent training runs with different random seeds. Since training the video model takes a considerable amount of time, we estimate the variance of our method through bootstrapping the validation dataset. We generate 5 validation datasets with 80% of the full size by sampling with replacement and use the quantitative results to estimate the standard deviation, reported in Table 13.

heading: B ADDITIONAL EXPERIMENTS AND RESULTS

text:
During the discussion period we have performed additional experiments based on the reviewers’ suggestions.
Additional Modality: Audio As our context adapter module is general in terms of modality, we have performed an additional experiment by using the audio signal that comes with the video. In this experiment (see Table 15) we use the audio encoder of GDT of Patrick et al. (2021) to encode five two second audio clips and add an MLP to map the audio embedding to the CLIP feature space.
We find that using comments alone is a stronger learning signal than audio alone. Exploiting both, comments and audio, improves the performance over using each auxiliary modality alone. Using audio alone only slightly improves R10 numbers upon not using any auxiliary modality. We hypothesise this is because the embedding learned by GDT’s audio encoder network is considerably different to the embedding learned by our pre-trained CLIP model. This insight is supported by the fact that fine-tuning the CLIP network together with the audio MLP is essential to learn anything in this setting.
Ablation Image Model: Frame Time For our image model, that operates on single frames only, we ablate the choice of frame within the video in Table 16.
We find that the choice of frame does indeed matter, with the middle frame in a video performing better than the first frame. This result is relatively intuitive since the actual content of a video usually happens in the middle of the video which is similar to the photographer bias (pictures tend to display an object in the middle of the frame).

heading: Model #Distr. TVR R@1 TVR R@10 VTR R@1 VTR R@10

text:
Distractor Comments In order to test the ability of our adapter to discard irrelevant information, we evaluated how our models perform when we progressively add distractor comments from other videos compared to the averaging baseline from Table 2. We find that, while both methods get gradually worse with more distractor comments, our method’s retrieval performance decreases at a slower rate, as it can be seen when looking at the percentage decrease from the original score, showing that it is better equipped to discard irrelevant information. It is important to note that the model was not trained with explicit distractors and has learned this capability through the builtin attention mechanism in the context adapter model.
Additional Dataset To test the performance of our model on another publicly available dataset, we perform experiments using the LiveBot Dataset (Ma et al., 2019). As our pre-trained backbone is trained only in English, but the dataset is collected from a Chinese website, we translate all title and comments to English before we run our method (see Table 18).
Consistent with the results in the main paper, the retrieval performance of CLIP is surpassed by our model trained with comments. Using comments during retrieval further improves the results, confirming that comments are a useful modality and that our approach can successfully leverage this auxiliary information.
Filtering Comments by Similarity To understand the importance of learning to filter comments vs. using a simple thresholding technique on the similarly returned by comparing embeddings, we conduct the following experiment in Table 19. We use the similarity between a title and a comment as computed by the encoder model and remove comments that differ from the title more than a certain threshold. Since this experiments evaluates a non-learned filtering procedure we use the averaging baseline from Table 2 as reference model.
We find that removing comments generally reduces the retrieval performance as comments can indeed contain additional information. Simply thresholding the similarity metric cannot distinguish between comments that have other but useful information and comments that do not contain information related to the video. Our model learns to filter comments and can thus make use of this information.
C MODEL DIAGRAM