text:
\section{Introduction}\label{sec:intro}

Humans communicate with each other in this interactive and dynamic visual world via languages, signs, and gestures.
The ability to jointly understand both visual and textual clues is an essential ability for intelligent agents to interpret multimodal signals in the physical world.
A wide range of tasks based on real-life videos have been designed to test such ability, including text-to-video retrieval~\cite{xu2016msr, krishna2017dense, rohrbach2015dataset}, video captioning~\cite{rohrbach2015dataset,xu2016msr,zhou2017towards}, video question answering~\cite{xu2017video, jang2017tgif, lei2018tvqa,lei2019tvqa+}, and video moment retrieval~\cite{anne2017localizing, gao2017tall, lei2020tvr}.
The \emph{de facto} paradigm to tackle these cross-modal tasks is to first extract dense video features from pre-trained vision models~\cite{he2016deep,carreira2017quo} and text features from pre-trained language models~\cite{pennington2014glove,devlin2018bert}, then apply multimodal fusion to wrangle together these fixed representations in a shared embedding space (Figure~\ref{fig:ClipBERT_comparison} (\textit{top})).

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth]{figs/ClipBERT_comparison.pdf}
  \caption{
Comparison between popular video-and-language learning paradigm (\textit{top}) and \textsc{ClipBERT}~(\textit{bottom}).
In contrast to most existing methods that utilize offline (\textit{stop gradient}) extracted \textit{dense video features} and \textit{text features}, \textsc{ClipBERT}~uses \textit{sparsely sampled clips} and \textit{raw text tokens} for \textit{end-to-end} modeling.
}
  \label{fig:ClipBERT_comparison}
  \vspace{-3mm}
\end{figure}

Existing approaches~\cite{jang2017tgif,lei2018tvqa,zhu2020actbert,le2020hierarchical} following this paradigm have achieved strong success, yet suffer from two main drawbacks: $(i)$ \emph{Disconnection in tasks/domains}: offline feature extractors are often trained on tasks and domains different from the target task. For example, features learned for action recognition from human activity videos~\cite{kay2017kinetics} are incongruently applied to downstream video question answering on generic-domain GIF videos~\cite{jang2017tgif}. $(ii)$ \emph{Disconnection in multimodal features}: learned features from different modalities are independent of each other. For instance, action recognition models~\cite{tran2015learning, wang2016temporal, carreira2017quo} are typically trained from pure video data without textual input, yet are applied to video-and-language tasks.
End-to-end task-specific finetuning offers a way to mitigate these inherent disconnections.
However, extracting features from the full sequence of video frames, as in most existing work, casts excessive demand on memory and computation, rendering it difficult or even infeasible to directly plug feature extractors into a video+language learning framework for efficient end-to-end finetuning.

Motivated by this, we propose \textsc{ClipBERT}, a generic and efficient framework for end-to-end video-and-language learning (Figure~\ref{fig:ClipBERT_comparison} (\textit{bottom})).
Two aspects distinguish \textsc{ClipBERT}~from previous work.
First, in contrast to densely extracting video features (adopted by most existing methods), \textsc{ClipBERT}~sparsely samples only one single or a few short clips from the full-length videos at each training step. The hypothesis is that visual features from sparse clips already capture key visual and semantic information in the video, as consecutive clips usually contain similar semantics from a continuous scene. Thus, a handful of clips are sufficient for training, instead of using the full video.
Then, predictions from multiple densely-sampled clips are aggregated to obtain the final video-level prediction during inference, which is less computational demanding.
This \emph{sparse-training}-then-\emph{dense-inference} strategy greatly reduces memory needs and computations, allowing economical end-to-end learning from raw video frame pixels and language tokens.

The second differentiating aspect concerns the initialization of model weights (\ie, transfer through pre-training).
In recent literature, image-text pre-training (\eg, using COCO Captions~\cite{chen2015microsoft} or Visual Genome Captions~\cite{krishna2017visual}) has been applied to image-text tasks~\cite{tan2019lxmert,lu2019vilbert,chen2019uniter,su2019vl,huang2020pixel,li2020unicoder,zhou2020unified}, and video-text pre-training (\eg, using HowTo100M~\cite{miech2019howto100m}) to video-related tasks~\cite{sun2019videobert,zhu2020actbert,gabeur2020multi,li2020hero}.
There has been no study to cross-examine the effect of image-text pre-training on video-text tasks. Intuitively, visual features learned through pre-training from large-scale image datasets should also help video understanding tasks that rely on visual clues in static video frames.
To investigate this, we use 2D architectures (\eg, ResNet-50~\cite{he2016deep}) instead of 3D features ~\cite{tran2015learning,carreira2017quo,qiu2017learning,Xie_2018_ECCV} as our visual backbone for video encoding, allowing us to harness the power of image-text pre-training for video-text understanding along with the advantages of low memory cost and runtime efficiency.
Empirically, we observe that the knowledge learned in image-text pre-training indeed helps video-text tasks; this simple strategy helps us achieve better or comparable performance to previous state of the art on text-to-video retrieval and video question answering tasks.

Our contributions are three-fold:
($i$) We propose \textsc{ClipBERT}, a new end-to-end learning framework for video+language tasks.
Experiments show that \textsc{ClipBERT}~achieves superior (or on par) performance than existing methods across diverse video-text tasks, where the average video length ranges from a few seconds to three minutes.
($ii$) Our work suggests \textit{``less is more''}: the proposed end-to-end training strategy with a single or a few (\textit{less}) sparsely sampled clips is often \textit{more} accurate than traditional approaches that employ densely extracted video features.
($iii$)  We demonstrate that image-text pre-training benefits video-text tasks.
We also provide comprehensive ablation studies to reveal the key factors that lead to the success of \textsc{ClipBERT}, in hope of inspiring more future work.

\section{Related Work}\label{sec:related_work}
\paragraph{Video and Language Understanding.}
Popular video-and-language tasks include text-to-video retrieval~\cite{xu2016msr, krishna2017dense, rohrbach2015dataset}, video captioning~\cite{xu2016msr, zhou2017towards, krishna2017dense, rohrbach2015dataset, li2016tgif}, video question answering~\cite{xu2017video, jang2017tgif, lei2018tvqa}, and moment retrieval~\cite{anne2017localizing, gao2017tall, lei2020tvr}.
Standard approaches~\cite{jang2017tgif, xu2017video, gao2018motion, zhang2018cross, lei2018tvqa, fan2019heterogeneous, le2020hierarchical,lei2020mart} leverage offline extracted video and text features from action recognition models~\cite{kay2017kinetics, wang2016temporal, carreira2017quo, Xie_2018_ECCV}, image recognition models~\cite{deng2009imagenet, he2016deep}, and language models~\cite{mikolov2013distributed, pennington2014glove, devlin2018bert, liu2019roberta}.
Aligned with the success of transformer-based~\cite{vaswani2017attention} language pre-training~\cite{devlin2018bert,liu2019roberta,yang2019xlnet,raffel2020exploring,lan2019albert,clark2020electra} and image-text pre-training~\cite{tan2019lxmert,lu2019vilbert,chen2019uniter,li2020unicoder,huang2020pixel,zhou2020unified,gan2020large,cho2021unifying},
video-text pre-training ~\cite{sun2019videobert,zhu2020actbert,gabeur2020multi,li2020hero,miech2020end,miech2019howto100m} has shown promising results on video-and-language tasks.
Beyond using fixed features and same-domain data (\ie, video-text pre-training only for video-text tasks), our work focuses on end-to-end training and applying image-text pre-training for video-text tasks.

\paragraph{Action Recognition.}
Modern video action recognition architectures are typically designed with deep 2D~\cite{simonyan2014very,szegedy2015going,he2016deep} or 3D~\cite{tran2015learning,carreira2017quo,Xie_2018_ECCV} convolutional networks.
These backbones are often computation and memory heavy, making it extremely difficult to directly process videos of considerable length.
To ease this difficulty, instead of training on full-length videos, models are often trained with randomly sampled short clips from the videos~\cite{simonyan2014two,tran2015learning,qiu2017learning,Xie_2018_ECCV,wang2018non,feichtenhofer2019slowfast,feichtenhofer2020x3d,wang2016temporal}.
At inference time, predictions from multiple uniformly sampled clips are aggregated together as the final video-level prediction.
In relation to these works, we adopt a similar strategy to perform sparse training and dense inference to reduce overhead on video processing, but focus on video-and-language tasks with cross-modal modeling of video and language, in contrast to pure video modeling.

\section{\textsc{ClipBERT}~with Sparse Sampling}
We propose \textsc{ClipBERT}, a general framework that enables end-to-end learning on video and language data, by learning joint representations directly from video frame pixels and raw text tokens, instead of from offline-extracted single-modality features.
Figure~\ref{fig:ClipBERT_comparison} (\textit{bottom}) gives an overview of \textsc{ClipBERT}~framework.
It adopts a sparse sampling strategy using only a single or a few sampled clips at each training step, instead of full-length videos.
Each sampled clip is independently encoded with a vision backbone model, the visual features from which are then fed to a cross-modal module that extracts relations between the clip and its associated text representations.
Independent predictions from all the sampled clips are fused together (\eg, \textit{through mean-pooling}) to derive a consensus at the video level.
A task-specific loss is calculated based on this consensus to learn model parameters.
During inference, \textsc{ClipBERT}~densely samples a sequence of clips from the video and aggregates their predictions as the final prediction.

Most existing work~\cite{jang2017tgif,lei2018tvqa,zhu2020actbert,le2020hierarchical} models offline-extracted dense video features and text features.
Formally, we denote a video-text pair as $V$ (for video) and $S$ (for text sequence). The video $V$ is further denoted as a list of $N$ clips of equal duration $[c_1, c_2, ..., c_N]$.
This standard paradigm can be formulated as:
\begin{align}\label{eq:existing_approach_modeling}
    p \,{=}\, \mathcal{H}([\mathcal{F}^{SG}_{v}(c_1),
    \mathcal{F}^{SG}_{v}(c_2),
    ..., \mathcal{F}^{SG}_{v}(c_N)],
    \mathcal{F}^{SG}_{l}(S)),
\end{align}
\noindent
where $\mathcal{F}^{SG}_{v}$ and $\mathcal{F}^{SG}_{l}$ are vision and language encoder, respectively. The superscript $SG$ denotes \textit{Stop Gradient}, meaning that gradients cannot be back-propagated through the two encoders.
$\mathcal{H}$ is a cross-modal encoder and predictor, which models the relations between the encoded video/language inputs and makes predictions.
$p$ is the video-level prediction. A task-specific loss function $\mathcal{L}^{task}$ is then applied to calculate the loss value $l^{task}$ based on this prediction and its corresponding ground-truth $q$:
\begin{align}\label{eq:existing_approach_loss}
    l^{task} = \mathcal{L}^{task}(p, q).
\end{align}

\paragraph{Sparse Sampling for Training.}
Instead of using the full-length video with $N$ clips, \textsc{ClipBERT}~sparsely (and randomly) samples $N_{train}$ clips $\{c_{\tau_{i}}\}_{i=1}^{N_{train}}$ from $V$ for training.
$N_{train}$ is typically much smaller than $N$.
We model a sampled clip $c_{\tau_{i}}$ together with text input $S$ to produce a prediction $p_{\tau_{i}}$:
\begin{align}\label{eq:clip_bert_modeling}
    p_{\tau_{i}} = \mathcal{H}(\mathcal{F}_{v}(c_{\tau_{i}}), \mathcal{F}_{l}(S)),
\end{align}

\noindent
where $\mathcal{F}_{v}$ and $\mathcal{F}_{l}$ are vision/language encoders.
Different from Equation~\ref{eq:existing_approach_modeling} that uses offline vision/language encoders, \textsc{ClipBERT}~is end-to-end trainable, allowing task-specific loss to further finetune the encoders, learning better representations.
Independent predictions from all sampled clips are aggregated to derive a consensus. The loss value $l^{task}$ is calculated based on this video-level consensus:
\begin{align}\label{eq:clip_bert_loss}
    l^{task} = \mathcal{L}^{task}(\mathcal{G}(p_{\tau_{1}}, p_{\tau_{2}}, ..., p_{\tau_{N_{train}}}), q),
\end{align}

\noindent
where $\mathcal{G}$ is the prediction/score aggregation function (\eg, mean-pooling).
When $N_{train}$ is larger than one, this formulation can be regarded as a form of multiple instance learning (MIL)~\cite{wu2015deep}.
At inference, we uniformly sample $N_{test}$ clips of the same duration as training clips, then aggregate predictions from all $N_{test}$ clips to form our final prediction.

\textsc{ClipBERT}'s sparse training strategy can be interpreted as a type of data augmentation: different subsets of clips from the same video are used at different training steps, which improves the model's generalization ability.
In this sense, it is analogous to random cropping~\cite{simonyan2014very,he2016deep} commonly used in image classification tasks.
It also takes inspiration from action recognition methods~\cite{simonyan2014two,tran2015learning,wang2016temporal,feichtenhofer2019slowfast}, where a video classifier is trained on sampled clips.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth]{figs/ClipBERT_instantiation.pdf}
  \vspace{-15pt}
  \caption{
Overview of \textsc{ClipBERT}~architecture. For simplicity, we only show an illustration of producing prediction for a single sampled clip. When multiple clips are used, their predictions are fused together as the final prediction.
}
  \label{fig:ClipBERT_instantiation}
  \vspace{-3mm}
\end{figure}

\paragraph{Model Architecture.}
Figure~\ref{fig:ClipBERT_instantiation} gives an overview of \textsc{ClipBERT}~architecture.
For the vision encoder $\mathcal{F}_{v}$, we use a 2D CNN architecture ResNet-50~\cite{he2016deep} instead of 3D architectures (such as C3D~\cite{tran2015learning} or I3D~\cite{carreira2017quo}), because 2D models typically consume less memory and run faster.
Besides, 2D CNNs have proved to work reasonably well on video understanding tasks such as action recognition~\cite{wang2016temporal,qiu2017learning}.
Specifically, we take the first 5 Conv blocks of ResNet-50~\cite{he2016deep} and add an extra convolution layer to reduce its output feature depth, as well as a 2\x2 max-pooling layer for spatial down-sampling, following Pixel-BERT~\cite{huang2020pixel}.
For each sampled clip, we uniformly sample $T$ frames and obtain $T$ feature maps.
A temporal fusion layer $\mathcal{M}$ (\eg, mean-pooling) is applied to aggregate the frame-level feature maps into a single clip-level feature map.
We then add a row-wise and a column-wise position embedding to each feature vector based on their 2D position.
These embeddings are the same trainable position embeddings as in BERT~\cite{devlin2018bert}.
Collectively, these two position embeddings are indicative of 2D spatial locations of the features, which can be viewed as a \textit{2D position embedding}.
The resulting feature map is flattened into an embedding sequence to represent the clip.

We use a trainable word embedding layer as our language encoder $\mathcal{F}_{l}$ to encode language tokens and add trainable position embeddings to encode positional information of the tokens.
Next, we add different type embeddings~\cite{devlin2018bert} to both clip and text embeddings to indicate their source type.
These two sequences are then concatenated as inputs to a 12-layer transformer~\cite{vaswani2017attention,devlin2018bert} for cross-modal fusion.
Special tokens \texttt{[CLS]} and \texttt{[SEP]} are added in this process following \cite{devlin2018bert}.
Given a downstream task, we add a task-specific prediction head with the last-layer \texttt{[CLS]} representation as input (\eg, using a two-layer MLP with softmax to produce scores for text-to-video retrieval).

\paragraph{Weight Initialization and Pre-training.}
We initialize the ResNet-50 layers with weights from grid-feat~\cite{jiang2020defense,ren2015faster}. It is trained on Visual Genome~\cite{krishna2017visual} for object detection and attribute classification, and produces effective grid features for image VQA tasks~\cite{antol2015vqa,gurari2018vizwiz}.
Input frames are resized to have a maximum longer side of $L$ while keeping the aspect ratios, and the shorter side is zero-padded to be $L$ as well~\cite{nguyen2020revisiting}.
We initialize the transformer and word embedding layers from BERT-base model~\cite{devlin2018bert}, pre-trained on BookCorpus~\cite{zhu2015aligning} and English Wikipedia.
These weights are trained separately for their individual single-modality tasks, thus simply combining them together in a single framework for downstream task training may result in sub-optimal performance.
Although pre-training the whole model end-to-end with large-scale video-text datasets such as HowTo100M~\cite{miech2019howto100m} are appealing, we are restricted by the enormous computation cost.\footnote{\cite{miech2020end} reports that pre-training I3D~\cite{carreira2017quo} with offline extracted text features on HowTo100M requires $\sim$3 days with 64 Cloud TPUs v3.
}
Luckily, as we use 2D CNN as our vision encoder, \textsc{ClipBERT}~is able to directly take image-text pairs as inputs for training.
Thus, we leverage large-scale image-text datasets (COCO Captions~\cite{chen2015microsoft} and Visual Genome Captions~\cite{krishna2017visual}) to perform cross-modal pre-training~\cite{tan2019lxmert,lu2019vilbert,huang2020pixel}.
Specifically, we use masked language modeling~\cite{devlin2018bert} and image-text matching~\cite{tan2019lxmert,lu2019vilbert} objectives to optimize the model.
By default, we finetune our model from this pre-trained weights for downstream video-text tasks.
The impact of different weight initialization strategies is examined in Section~\ref{subsec:analysis_pretraining_and_e2e_training}.

\paragraph{Implementation Details.}
We perform image-text pre-training on COCO Captions~\cite{chen2015microsoft} and Visual Genome Captions~\cite{krishna2017visual}.
These two datasets contain a total of 5.6M training image-text pairs on 151K images.
This is the same data used in UNITER's~\cite{chen2019uniter} in-domain pre-training.
We use input image size $L{=}768$, and the resulting feature map from the vision encoder contains 144 pixels.
To improve generalization and reduce computation cost, during pre-training, we follow Pixel-BERT~\cite{huang2020pixel} to use pixel random sampling that samples 100 pixels from the encoded feature map as the input to the transformer layers.
Note that we only apply pixel random sampling for pre-training, and always use the full feature map for downstream tasks to avoid misalignment in training and inference~\cite{huang2020pixel}.
We use WordPiece embeddings~\cite{wu2016google} and keep at most 20 tokens from the caption.
We then randomly mask 15\% of the tokens for masked language modeling.
For each image-caption pair, with a probability of 0.5, we replace the ground-truth caption with a randomly sampled caption from another image to form a negative pair for image-text matching.
We use AadmW~\cite{loshchilov2017decoupled} to optimize end-to-end model training, with an initial learning rate of 5e-5, $\beta_1{=}0.9$, $\beta_2{=}0.98$, and use learning rate warmup over the first 10\% training steps followed by linear decay to 0.
Our model is implemented in PyTorch~\cite{paszke2019pytorch} and transformers~\cite{wolf-etal-2020-transformers}.
It is trained for 40 epochs with mixed precision, on 8 NVIDIA V100 GPUs with a batch size of 32 per GPU. The whole training process takes 4 days to complete.

For downstream finetuning,
we use the same training and optimizer configurations except that the default input image size is set to 448 (due to the typically lower resolution of videos compared to images). Since downstream datasets vary in scale and domain, we use task-specific learning rates and training epochs based on validation performance.

\section{Experiments}\label{sec:experiments}
In this section, we evaluate \textsc{ClipBERT}~on two popular video-and-language tasks, text-to-video retrieval and video question answering, across six different datasets.
We also provide extensive ablation studies to analyze the key factors that contribute to \textsc{ClipBERT}'s success.

\subsection{Downstream Tasks}
\noindent\textbf{Text-to-Video Retrieval.}
$(i)$ \textbf{MSRVTT}~\cite{xu2016msr} contains 10K YouTube videos with 200K descriptions.
We follow~\cite{yu2018joint,miech2019howto100m}, using 7k train+val videos for training and report results on the 1K test set~\cite{yu2018joint}.
We also create a local val set by sampling 1K video-caption pairs from unused test videos for our ablation study.
$(ii)$ \textbf{DiDeMo}~\cite{anne2017localizing} contains 10K Flickr videos annotated with 40K sentences.
$(iii)$ \textbf{ActivityNet Captions}~\cite{krishna2017dense} contains 20K YouTube videos annotated with 100K sentences.
The training set contains 10K videos, and we use val1 set with 4.9K videos to report results.
For MSRVTT, we evaluate standard sentence-to-video retrieval. For DiDeMo and ActivityNet Captions, we follow~\cite{zhang2018cross, liu2019use} to evaluate paragraph-to-video retrieval, where all sentence descriptions for a video are concatenated to form a paragraph for retrieval. We use average recall at K (R@K) and median rank (MdR) to measure performance.

\noindent\textbf{Video Question Answering.}
$(i)$ \textbf{TGIF-QA}~\cite{jang2017tgif} contains 165K QA pairs on 72K GIF videos.
We experiment with 3 TGIF-QA tasks: \textit{Repeating Action} and \textit{State Transition} for multiple-choice QA, and \textit{Frame QA} for open-ended QA.
We leave the \textit{Count} task as future work as it requires directly modeling full-length videos.
$(ii)$ \textbf{MSRVTT-QA}~\cite{xu2017video} is created based on videos and captions in MSRVTT, containing 10K videos and 243K open-ended questions.
$(iii)$ \textbf{MSRVTT multiple-choice test}~\cite{yu2018joint} is a multiple-choice task with videos as questions, and captions as answers.
Each video contains 5 captions, with only one positive match.
For all the QA tasks, we use standard train/val/test splits and report accuracy to measure performance.

\begin{figure}[!t]
  \centering
  \vspace{-5pt}
  \includegraphics[width=\linewidth]{figs/video_duration.pdf}
  \vspace{-15pt}
  \caption{
Average video length in different datasets.
}
  \label{fig:video_duration}
\end{figure}

Figure~\ref{fig:video_duration} shows a comparison of average video length of evaluated datasets.
Videos across datasets demonstrate considerable difference in domains and lengths, ranging from 3-second generic-domain GIF videos in TGIF-QA to 180-second human activity videos in ActivityNet Captions.

\begin{table}[!t]
\tablestyle{5pt}{1.1}
\def19pt{20pt}
\scalebox{1.0}{
\begin{tabular}{c|R{14pt}R{19pt}R{19pt}R{19pt}|c}
\multirow{2}{*}{ $L$ } & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\  \cline{2-5}
& R1 & R5 & R10 & MdR & QA Acc. \\
\shline
224 & 6.8 & 24.4 & 35.8 & 20.0 & \bf 35.78 \\
448 & \bf 10.2 & \bf 28.6 & \bf 40.5 & \bf 17.0 & \bf  35.73 \\
768 & \bf 11.0 & 27.8 & \bf 40.9 & \bf 16.0 & \bf 35.73 \\
1000 & 10.0 & \bf 28.4 & 39.4 & 18.0 & 35.19 \\
\end{tabular}
}
\caption{
Impact of \textbf{input image size $L$}.
}
\label{tab:image_size}
\vspace{-3mm}
\end{table}

\subsection{Analysis of Sparse Sampling}\label{subsec:analysis_sparse_sampling}
We conduct comprehensive ablation studies concerning various aspects of \textsc{ClipBERT}'s design in this section and Section~\ref{subsec:analysis_pretraining_and_e2e_training}.
If not otherwise stated, we randomly sample a single frame ($N_{train}{=}1$ and $T{=}1$) from full-length videos for training, and use the middle frame ($N_{test}{=}1$) for inference, with input image size $L{=}448$.
All ablation results are on MSRVTT retrieval local val and MSRVTT-QA val sets.

\paragraph{Do we need larger input image size?}
We compare models with different input image sizes $L \in \{224, 448, 768, 1000\}$, results shown in Table~\ref{tab:image_size}.
Compared to the model with $L{=}224$, larger input resolution improves performance on the retrieval task, while maintaining a similar performance on the QA task.
The best performance is achieved at around $L{=}448$. Further increasing the resolution does not provide significant performance boost.
\cite{jiang2020defense} shows that increasing input image size from 448 to 1333 always improves image VQA~\cite{antol2015vqa} performance with no sign of saturation, while we observe the performance converges at around 448 for MSRVTT retrieval and QA.
This is potentially because VQA images are typically of higher raw resolution than MSRVTT videos (we are only able to obtain videos at a maximum height of 240 pixels).
We expect higher resolution videos could further improve model performance.

\paragraph{Do we need densely sampled frames?}
A common practice for video understanding and video+language understanding is to model densely sampled frames from the original video
(\eg,~\cite{carreira2017quo,Xie_2018_ECCV} sample frames at 25 frames per second).
To understand the impact of using densely sampled frames, we conduct a set of controlled experiments.
Specifically, we randomly sample a fixed-length 1-second clip from the video, then evenly sample $T {=} \{1, 2, 4, 8, 16\}$ frames within this clip for training.
For inference, we use the middle clip of the video.
When multiple frames are used (\ie, $T$\textgreater1), we use mean pooling for temporal fusion.

We also experiment with variants using additional 3D convolutions before mean pooling: ($i$) Conv3D: a standard 3D convolution layer with kernel size 3, stride 1; ($ii$) Conv(2+1)D: a spatial and temporal separable 3D convolution~\cite{tran2018closer,Xie_2018_ECCV}.
Adding 3D convolutions to 2D convolutions essentially leads to a design similar to Top-Heavy S3D architecture~\cite{Xie_2018_ECCV}, which shows better performance than full 3D convolutions on video action recognition and runs faster.

Results are shown in Table~\ref{tab:num_frames_in_training}.
Overall, models that use 3D convolutions perform worse than models that adopt a simple mean pooling.
For mean pooling, we observe that using two frames provides a notable improvement over using a single frame.
However, models that use more than two frames perform similarly compared to the one using two frames, suggesting that two frames already represent enough local temporal information for the tasks.

\begin{table}[!t]
\tablestyle{5pt}{1.1}
\def19pt{20pt}
\scalebox{1.0}{
\begin{tabular}{c|c|R{12pt}R{19pt}R{19pt}R{19pt}|c}
\multirow{2}{*}{$\mathcal{M}$} & \multirow{2}{*}{$T$} & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\ \cline{3-6}
 &  & R1 & R5 & R10 & MdR & QA Acc. \\
\shline
- & 1 & 10.2 & 28.6 & 40.5 & 17.0 & 35.73 \\
\hline
\multirow{4}{*}{ Mean Pooling } & 2 & \bf 11.3 & 31.7 & 44.9 & 14.0 & \bf 36.02 \\
& 4 & 10.8 & 30.0 & 43.6 & 14.0  & 35.83 \\
& 8 & 10.6 & \bf 32.5 & \bf 45.0 & \bf 13.0 & 35.69 \\
& 16 & \bf 11.6 & \bf 33.9 & \bf 45.8 & \bf 13.0 & \bf 36.05 \\
\hline
\multirow{2}{*}{ Conv3D } & 2 & 8.7 & 27.3 & 40.2 & 17.0 & 34.85 \\
& 16 & 10.1 & 28.9 & 41.7 & 16.0 & 35.03 \\
\hline
\multirow{2}{*}{ Conv(2+1)D } & 2 & 7.3 & 24.1 & 35.6 & 22.0 & 34.13 \\
& 16 & 9.9 & 27.3 & 39.6 & 17.0 & 33.92 \\
\end{tabular}
}
\caption{
Impact of \textbf{\#frames ($T$) and temporal fusion function ($\mathcal{M}$).}
We use a 1-second clip for all experiments.
}
\label{tab:num_frames_in_training}

\end{table}

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth]{figs/num_infer_clips.pdf}
  \caption{
Impact of \textbf{\#inference clips ($N_{test}$).}
}
  \label{fig:num_clips_at_inferece}
  \vspace{-3mm}
\end{figure}

\paragraph{Do more clips at inference help?}
At inference, we aggregate prediction scores from multiple densely sampled clips as the final score.
To show how this strategy affects performance, we evenly sample $N_{test} \in \{1, 2, 4, 8, 16\}$ clips from a video and average their individual predictions at inference.
For this experiment, we provide two models trained with different numbers of training frames per clip: one with a single frame and the other with two frames.
Both models use a single clip for training.
Results are shown in Figure~\ref{fig:num_clips_at_inferece}.
Adding more clips generally improves performance, especially the first few additions, but after a certain point performance saturates.
For example, in Figure~\ref{fig:num_clips_at_inferece} (\textit{left}), MSRVTT retrieval performance increases substantially as we use two and four clips, compared to using a single clip; then the performance gain gradually becomes marginal.

\begin{table}[!t]
\tablestyle{4pt}{1.1}
\def19pt{19pt}
\scalebox{1.0}{
\begin{tabular}{c|c|R{12pt}R{19pt}R{19pt}R{19pt}|c}
\multirow{2}{*}{$\mathcal{G}$} & \multirow{2}{*}{$N_{train}$} & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\ \cline{3-6}
 &  & R1 & R5 & R10 & MdR & QA Acc. \\
\shline
-  & 1 & 12.7 & 34.5 & 48.8 & 11.0 & 36.24 \\
\hline
\multirow{4}{*}{ Mean Pooling } & 2 & 13.3 & 37.1 & 50.6 & 10.0 & 35.94 \\
& 4 & 14.0 & 38.6 & 51.6 & 10.0 &  35.40 \\
& 8 & 13.4 & 36.4 & 49.7 & 11.0 &  35.76 \\
& 16 & 15.2 & \bf 39.4 & \bf 53.1 & \bf 9.0 & 35.33 \\
\hline
\multirow{2}{*}{ Max Pooling } & 2 & 8.5 & 28.7 & 42.2 & 14.0 & \bf 36.41 \\
& 16 & 12.5 & 33.1 & 46.8 & 12.0 & 36.25 \\
\hline
\multirow{2}{*}{ LogSumExp } & 2 & \bf 15.5 & 38.4 & 52.6 & \bf 9.0 & \bf 36.59 \\
& 16 & \bf 17.4 & \bf 41.5 & \bf 55.5 & \bf 8.0 & 36.16 \\
\end{tabular}
}
\caption{
Impact of \textbf{\#training clips ($N_{train}$) and score aggregation function ($\mathcal{G}$).}
All models use $N_{test}{=}$16 clips for inference.
}
\label{tab:num_clips_for_training}
\vspace{-3mm}
\end{table}

\begin{table}[!t]
\tablestyle{2pt}{1.1}
\def19pt{19pt}
\scalebox{1.0}{
\begin{tabular}{c|c|R{12pt}R{19pt}R{19pt}R{19pt}|c}
\multirow{2}{*}{Sampling Method} & \multirow{2}{*}{$N_{train}$} & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\ \cline{3-6}
 &  & R1 & R5 & R10 & MdR & QA Acc. \\
\shline
\multirow{1}{*}{ Dense Uniform } & 16 & 15.5 & 39.6 & 55.0 & 9.0 & 35.88 \\
\hline
\multirow{3}{*}{ Sparse Random } & 1 & 12.7 & 34.5 & 48.8 & 11.0 & 36.24 \\
 & 2 & 15.5 & 38.4 & 52.6 & 9.0 & 36.59 \\
& 4 & \bf 15.7 & \bf 41.9 & \bf 55.3 & \bf 8.0 & \bf 36.67 \\
\end{tabular}
}
\caption{
\textbf{Sparse random sampling \textit{vs.} dense uniform sampling.}
All models use $N_{test}{=}$16 clips for inference.
}
\label{tab:dense_uniform_vs_sparse_random}
\vspace{-3mm}
\end{table}

\paragraph{Do more clips in training help?}
We randomly sample $N_{train}$ clips and aggregate scores from the clips with aggregation function $\mathcal{G}$ as the final score to calculate the training loss.
When multiple clips are used, information from these samples is aggregated through multiple instance learning to maximize the utility of these clips.
To understand how this strategy affects model performance, we evaluate model variants that use $N_{train} \in \{1, 2, 4, 8, 16\}$ at training.
We also consider 3 different commonly used score aggregation functions for $\mathcal{G}$: mean pooling, max pooling, and LogSumExp~\cite{miech2020end}. In mean pooling and max pooling, the cross-clip pooling is performed over logits, followed by a softmax operator. In LogSumExp, logits from each clip are first fed through an element-wise exponential operator, followed by a cross-clip mean pooling. The aggregated output is further normalized by its own sum to make it eligible as a probability distribution. For simplicity, we always use the same aggregation function for training and inference.
For a fair comparison, all models use a single frame per clip for training and 16 clips for inference, \ie, $T{=}1$ and $N_{test}{=}16$.

Results are shown in Table~\ref{tab:num_clips_for_training}.
In general, adding more clips helps, and the second added clip gives the most performance gain.
For example, for models with LogSumExp, $N_{train}{=}2$ improves retrieval R1 score of $N_{train}{=}1$ by 2.8\%, while $N_{train}{=}16$ improves only 1.9\% upon $N_{train}{=}2$, even though it adds much more clips.
As for score aggregation function $\mathcal{G}$, LogSumExp works the best.

\begin{figure}[!t]
  \centering
  \includegraphics[width=\linewidth]{figs/memory_computation_cost.pdf}
  \vspace{-3mm}
  \caption{
\textbf{Memory and computation cost comparison} w.r.t. different numbers of clips ($N_{train}$) or frames ($T$) at training.
\textbf{(a)}: Maximum allowed batch size with fixed $T{=}1$.
\textbf{(b)}: Time cost for a single forward and backward pass with fixed $T{=}1$, batch size 8.
\textbf{(c)}: Maximum allowed batch size with fixed $N_{train}{=}1$.
\textbf{(d)}: Time cost for a single forward and backward pass with fixed $N_{train}{=}1$, batch size 8.
All experiments are conducted on a single NVIDIA V100 GPU with 32GB memory.
MSRVTT retrieval performance is also added in (b, d) for reference. Best viewed in color.
}
  \label{fig:memory_computation_cost}
  \vspace{-3mm}
\end{figure}

\paragraph{Sparse Random Sampling \textit{vs.} Dense Uniform Sampling.}
At each training step, \textsc{ClipBERT}~randomly samples only a single or a few short clips from a full-length video.
Intuitively, this sparse random sampling strategy can be interpreted as a type of data augmentation where different subsets of clips for a video are used to calculate the loss at different training steps.
To show the effectiveness of this approach, we compare \textsc{ClipBERT}~with a variant that uses uniformly sampled dense clips.
Specifically, we use the same \textsc{ClipBERT}~architecture as before but always uses 16 uniformly sampled clips for training.
Table~\ref{tab:dense_uniform_vs_sparse_random} shows the comparison.
Sparse random sampling with only 4 clips outperforms dense uniformly sampling with 16 clips across all metrics in both retrieval and QA tasks.
Meanwhile, using 4 clips is much more memory and computation efficient than using 16 clips, as we show in the next paragraph.
In addition to these two sampling approaches, it is also possible to choose clips using content-based methods such as~\cite{wu2019adaframe}.
However, this requires an extra non-trivial selection step, and may also remove some of the data augmentation effect brought by random sampling.

\paragraph{Memory and Computation Cost.}
Figure~\ref{fig:memory_computation_cost} shows a comparison of memory and computation cost \emph{w.r.t.} different numbers of clips ($N_{train}$) or frames ($T$) at training.
We observe that using more clips or more frames at training considerably increases memory demand and computational cost.
For example, in Figure~\ref{fig:memory_computation_cost}~(\textit{a}), we see that the maximum allowed batch size for a single NVIDIA V100 GPU is 190 when $N_{train}{=}2$, compared to that of 16 when $N_{train}{=}16$.
Meanwhile, in Figure~\ref{fig:memory_computation_cost}~(\textit{b}), we see that the time cost increases almost linearly with $N_{train}$, while the performance improvement on MSRVTT retrieval is less significant.
These comparisons demonstrate the efficiency and effectiveness of the proposed sparse training strategy.

\subsection{Analysis of Pre-training/End-to-end Training}\label{subsec:analysis_pretraining_and_e2e_training}

\begin{table}[!t]
\tablestyle{5pt}{1.1}
\scalebox{1.0}{
\begin{tabular}{l|l|R{12pt}R{11pt}R{11pt}R{20pt}|c}
\multicolumn{2}{c|}{ Weight Initialization } & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\  \cline{1-6}
CNN & transformer & R1 & R5 & R10 & MdR & QA Acc. \\
\shline
random & random & 0.3 & 0.4 & 0.9 & 506.0 & 28.05 \\
random & $\text{BERT}_{\textsc{base}}$ & 0.0 & 0.2 & 0.7 & 505.0 & 31.72 \\
TSN, K700 & $\text{BERT}_{\textsc{base}}$ & 5.7 & 22.1 & 33.1 & 23.0 & 35.40 \\
ImageNet & $\text{BERT}_{\textsc{base}}$  & 7.2 & 23.3 & 35.6 & 21.0 & 35.01 \\
grid-feat & $\text{BERT}_{\textsc{base}}$  & 7.4 & 21.0 & 30.7 & 26.0 & 35.27 \\
\hline
\multicolumn{2}{c|}{ image-text pre-training } & \bf 10.2 & \bf 28.6 & \bf 40.5 & \bf 17.0 & \bf 35.73 \\
\end{tabular}
}
\caption{
Impact of \textbf{weight initialization strategy}.
}
\label{tab:weight_init}
\vspace{-3mm}
\end{table}

\paragraph{Impact of Image-text Pre-training.}
Our model is initialized with image-text pre-training on COCO and Visual Genome Captions, to obtain better-aligned visual and textual representations.
To validate the effectiveness of using image-text pre-training for weight initialization, we also evaluate variants that use other weight initialization strategies.
Specifically, for CNN, we use weights from random initialization, image classification model pre-trained on ImageNet~\cite{deng2009imagenet}, frame-wise action recognition model TSN~\cite{wang2016temporal,2020mmaction2} pre-trained on Kinetics-700~\cite{smaira2020short,carreira2017quo}, or detection model grid-feat~\cite{jiang2020defense} pre-trained on Visual Genome~\cite{krishna2017visual}.
For transformer and word embedding layers, we use weights from random initialization or pre-trained $\text{BERT}_{\textsc{base}}$ model~\cite{devlin2018bert}.
For random initialization, we use the default setup in PyTorch~\cite{paszke2019pytorch} and Transformer~\cite{Wolf2019HuggingFacesTS} libraries for CNN and transformer layers, respectively.
Results are summarized in Table~\ref{tab:weight_init}.
We notice that randomly initializing CNN leads to massive performance degradation or even training failure, we hypothesize that it is mostly because of the difficulty of training large models on relatively small datasets (\eg, MSRVTT retrieval train split: 7K videos).
The best performance is achieved using image-text pre-trained weights, clearly indicating the benefit of utilizing image-text pre-training for video-text tasks.

\paragraph{Impact of End-to-End Training.}
The standard paradigm for video-and-language understanding is to train models with offline extracted features, in which the task objective does not affect the video and text encoding process.
In this work, we train our model in an end-to-end manner, allowing the model to finetune feature representations by leveraging task supervision.
In Table~\ref{tab:e2e}, we compare our model with variants that freeze portions of the parameters.
Overall, the best performance is achieved by our model, showing the importance of end-to-end training.
Note that all the models in Table~\ref{tab:e2e} are finetuned from our end-to-end image-text  pre-trained model, which partly resolves the multimodal feature disconnection issue in Section~\ref{sec:intro}.
Thus, we expect smaller improvement from further end-to-end finetuning.

\paragraph{Main Conclusions} from the analyses in Section~\ref{subsec:analysis_sparse_sampling} and Section~\ref{subsec:analysis_pretraining_and_e2e_training} are summarized as follows:
($i$) Larger input image size helps improve model performance, but the gain diminishes when image size is larger than 448;
($ii$) Sparsely sampling 2 frames from each clip performs on par with dense sampling 16 frames, showing that just one or two frames are sufficient for effective video-and-language training;
mean-pooling is more effective than 3D Conv when fusing information from different frames;
($iii$) More clips at inference helps improve model performance; aggregation strategy of predictions across clips affects final performance;
($iv$) Sparse (random) sampling is more effective than dense uniform sampling while being more memory and computation efficient;
($v$) Image-text pre-training benefits video-text tasks;
and ($vi$) End-to-end training improves model performance.

\begin{table}[!t]
\tablestyle{2pt}{1.1}
\def19pt{20pt}
\scalebox{1.0}{
\begin{tabular}{c|c|R{12pt}R{16pt}R{18pt}R{19pt}|c}
\multicolumn{2}{c|}{ Parameters Trainable? } & \multicolumn{4}{c|}{ MSRVTT Retrieval } & MSRVTT- \\ \cline{1-6}
\quad\;$\mathcal{F}_{v}$\quad\; & $\mathcal{F}_{l}$ & R1 & R5 & R10 & MdR & QA Acc. \\
\shline
\xmark & \xmark & 8.0 & 27.2 & 38.9 & 17.0 & \bf 35.78 \\
\xmark & \cmark & 9.0 & 27.5 & 39.4 & 18.0 & 35.50 \\
\hline
\cmark & \cmark & \bf 10.2 & \bf 28.6 & \bf 40.5 & \bf 17.0 & 35.73 \\
\end{tabular}
}
\caption{
Impact of \textbf{end-to-end training}.
}
\label{tab:e2e}
\vspace{-3mm}
\end{table}

\begin{table*}[t!]\centering
\subfloat[MSRVTT 1K test set.
\label{tab:main_msrvtt_retrieval}]{\tablestyle{2pt}{1.1}
\def19pt{15pt}
\begin{tabular}{l|R{14pt}R{19pt}R{19pt}r}
\multicolumn{1}{c|}{Method} & R1 & R5 & R10 & MdR \\
\shline
\demph{HERO~\cite{li2020hero} ASR, PT} & \demph{20.5} & \demph{47.6} & \demph{60.9} & \demph{-} \\
\hline
JSFusion~\cite{yu2018joint} & 10.2 & 31.2 & 43.2 & 13.0 \\
HT~\cite{miech2019howto100m} PT & 14.9 & 40.2 & 52.8 & 9.0 \\
ActBERT~\cite{zhu2020actbert} PT & 16.3 & 42.8 & 56.9 & 10.0 \\
HERO~\cite{li2020hero} PT & 16.8 & 43.4 & \bf 57.7 & - \\
\hline
\textsc{ClipBERT}~4\x1 & \bf 19.8 & \bf 45.1 & 57.5 & \bf 7.0 \\
\textsc{ClipBERT}~8\x2 & \bf 22.0 & \bf 46.8 & \bf 59.9 & \bf 6.0 \\
\multicolumn{5}{c}{} \\
\end{tabular}
}
\hspace{1mm}
\subfloat[DiDeMo test set.  \label{tab:main_didemo_retrieval}]{\tablestyle{2pt}{1.1}
\def19pt{15pt}
\begin{tabular}{l|R{14pt}R{19pt}R{19pt}r}
\multicolumn{1}{c|}{Method} & R1 & R5 & R10 & MdR  \\
\shline
\demph{CE~\cite{liu2019use}} & \demph{16.1}	& \demph{41.1} & \demph{-} & \demph{8.3}  \\
\hline
S2VT~\cite{venugopalan2014translating} & 11.9 & 33.6 & - & 13.0 \\
FSE~\cite{zhang2018cross}	& 13.9	& 36.0 & - & 11.0 \\
\hline
\textsc{ClipBERT}~4\x1 & \bf 19.9 & \bf 44.5 & \bf 56.7 & \bf 7.0  \\
\textsc{ClipBERT}~8\x2 &\bf 20.4	& \bf 48.0	& \bf 60.8	& \bf 6.0\\
\multicolumn{5}{c}{} \\
\multicolumn{5}{c}{} \\
\multicolumn{5}{c}{} \\
\end{tabular}
}
\hspace{1mm}
\subfloat[ActivityNet Captions val1 set.
\label{tab:main_anet_cap_retrieval}]{\tablestyle{2pt}{1.1}
\def19pt{15pt}
\begin{tabular}{l|R{14pt}R{19pt}R{19pt}r}
\multicolumn{1}{c|}{Method} & R1 & R5 & R10 & MdR  \\
\shline
\demph{CE~\cite{liu2019use}} & \demph{18.2}	& \demph{47.7} & \demph{-} & \demph{6.0}  \\
\demph{MMT~\cite{gabeur2020multi}} & \demph{22.7} & \demph{54.2} & \demph{93.2} & \demph{5.0}  \\
\demph{MMT~\cite{gabeur2020multi} PT} & \demph{28.7} & \demph{61.4} & \demph{94.5} & \demph{3.3}  \\
\hline
Dense~\cite{krishna2017dense} & 14.0 & 32.0 & - & 34.0 \\
FSE~\cite{zhang2018cross}	& 18.2	& 44.8 & - & 7.0 \\
HSE~\cite{zhang2018cross}	& 20.5	& \bf 49.3 & - & - \\
\hline
\textsc{ClipBERT}~4\x2$^*$ & \bf 20.9 & 48.6 & \bf 62.8 & \bf 6.0 \\
\textsc{ClipBERT}~4\x2$^*$ ($N_{test}{=}$20)  & \bf 21.3 & \bf 49.0 & \bf 63.5 & \bf 6.0 \\
\end{tabular}
}
\caption{
\textbf{Comparison with state-of-the-art methods on text-to-video retrieval}.
\textsc{ClipBERT}~models with different training input sampling methods are denoted by $N_{train}$\x$T$.
We use $N_{test}{=}16$ if not otherwise stated.
We gray out models that used features other than appearance and motion for a fair comparison, \eg, CE used appearance, scene, motion, face, audio, OCR, ASR features from 11 different models.
\textit{PT} indicates the model is pre-trained on HowTo100M. * denotes models use 2-second clips instead of the default 1-second clips.
\label{tab:main_retrieval}
}
\vspace{-3mm}
\end{table*}

\begin{table*}[!t]
\subfloat[TGIF-QA test set.
\label{tab:main_tgif_qa}]{\tablestyle{3pt}{1.1}
\def19pt{30pt}
\begin{tabular}{l|C{24pt}C{19pt}C{19pt}}
\multicolumn{1}{c|}{Method} & Action & Transition & FrameQA \\
\shline
ST-VQA~\cite{jang2017tgif} & 60.8 & 67.1 & 49.3 \\
Co-Memory~\cite{gao2018motion} & 68.2 & 74.3 & 51.5 \\
PSAC~\cite{li2019beyond} & 70.4 & 76.9 & 55.7 \\
Heterogeneous Memory~\cite{fan2019heterogeneous} & 73.9 & 77.8 & 53.8 \\
HCRN~\cite{le2020hierarchical} & 75.0 & 81.4 & 55.9 \\
QueST~\cite{jiang2020divide} & 75.9 & 81.0 & \bf 59.7 \\
\hline
\textsc{ClipBERT}~1\x1 ($N_{test}{=}$1) & \bf 82.9 & \bf 87.5 & 59.4 \\
\textsc{ClipBERT}~1\x1 & \bf 82.8 & \bf 87.8 & \bf 60.3 \\
\end{tabular}
}
\hspace{2mm}
\subfloat[MRSVTT-QA test set.
\label{tab:main_msrvtt_qa}]{\tablestyle{3pt}{1.1}
\def19pt{40pt}
\begin{tabular}{l|C{30pt}}
\multicolumn{1}{c|}{Method} & Accuracy \\
\shline
ST-VQA~\cite{jang2017tgif} (by \cite{fan2019heterogeneous}) & 30.9  \\
Co-Memory~\cite{gao2018motion} (by \cite{fan2019heterogeneous}) & 32.0 \\
AMU~\cite{xu2017video} & 32.5 \\
Heterogeneous Memory~\cite{fan2019heterogeneous} & 33.0 \\
HCRN~\cite{le2020hierarchical} & 35.6 \\
\hline
\textsc{ClipBERT}~4\x1 & \bf 37.0 \\
\textsc{ClipBERT}~8\x2 & \bf 37.4 \\
\multicolumn{2}{c}{} \\
\end{tabular}
}
\hspace{2mm}
\subfloat[MRSVTT multiple-choice test. \label{tab:main_msrvtt_multiple_choice_test}]{\tablestyle{3pt}{1.1}
\def19pt{40pt}
\begin{tabular}{l|C{30pt}}
\multicolumn{1}{c|}{Method} & Accuracy \\
\shline
SNUVL~\cite{yu2016video} (by \cite{yu2018joint})  & 65.4 \\
ST-VQA~\cite{jang2017tgif} (by \cite{yu2018joint}) & 66.1 \\
CT-SAN~\cite{yu2017end} (by \cite{yu2018joint}) & 66.4 \\
MLB~\cite{kim2016hadamard} (by \cite{yu2018joint}) & 76.1 \\
JSFusion~\cite{yu2018joint} & 83.4 \\
ActBERT~\cite{zhu2020actbert} PT & 85.7 \\
\hline
\textsc{ClipBERT}~4\x1 & \bf 87.9 \\
\textsc{ClipBERT}~8\x2 & \bf 88.2 \\
\end{tabular}
}\caption{
\textbf{Comparison with state-of-the-art methods on video question answering}.
}
\label{tab:main_video_qa}
\end{table*}

\subsection{Comparison to State-of-the-art}
For evaluation, we compare \textsc{ClipBERT}~with state-of-the-art methods on text-to-video retrieval and video question answering tasks.
We denote models using different sampling methods at training as \textit{\textsc{ClipBERT}~$N_{train}$\x$T$}, (randomly sample $N_{train}$ 1-second clips for training, each contains $T$ uniformly sampled frames of size $L{=}448$).
We use LogSumExp to aggregate scores from multiple clips.
At inference time, if not otherwise stated, we aggregate scores from $N_{test}{=}$16 uniformly sampled clips.

\paragraph{Text-to-Video Retrieval.}
Table~\ref{tab:main_retrieval} summarizes results on text-to-video retrieval.
In Table~\ref{tab:main_msrvtt_retrieval}, \textsc{ClipBERT}~achieves significant performance gain over existing methods on MSRVTT retrieval, including HT~\cite{miech2019howto100m}, ActBERT~\cite{zhu2020actbert}, and HERO~\cite{li2020hero}, which are pre-trained on 136M clip-caption pairs from HowTo100M~\cite{miech2019howto100m}.
Under a fair comparison, \textsc{ClipBERT}~4\x1 outperforms HERO~\cite{li2020hero} by 3.0\% on R@1.
Note that HERO uses SlowFast~\cite{feichtenhofer2019slowfast} features extracted from full-length videos at a very dense frame rate of 21 FPS (\ie, on average 310 frames per video for MSRVTT), while \textsc{ClipBERT}~4\x1 uses only 4 randomly sampled frames.
When more frames are used, \textsc{ClipBERT}~8\x2 achieves even higher performance, surpassing HERO by 5.2\%.
Compared to the HERO ASR model that uses extra input from Automatic Speech Recognition (ASR), \textsc{ClipBERT}~still obtains 1.5\% higher R1 score.

Similarly, on DiDeMo and ActivityNet Captions paragraph-to-video retrieval tasks (Table~\ref{tab:main_didemo_retrieval} and Table~\ref{tab:main_anet_cap_retrieval}), we notice our best \textsc{ClipBERT}~models outputform CE~\cite{liu2019use} by 4.3\% and 3.1\% on R1, respectively, despite CE's use of appearance, scene, motion, face, audio, OCR, ASR features densely extracted from 11 different models.
ActivityNet Caption videos are on average 180-second long.
In Table~\ref{tab:main_anet_cap_retrieval} we show \textsc{ClipBERT}~performs competitively with existing methods that model long-range relations in this dataset.
Especially, \textsc{ClipBERT}~obtains 0.8\% higher R1 than HSE~\cite{zhang2018cross} and is competitive compared to MMT~\cite{gabeur2020multi} that uses extra audio features\footnote{\cite{gabeur2020multi} shows that adding audio features greatly improves performance.}, even though \textsc{ClipBERT}~4\x2$^*$ ($N_{test}{=}20$) samples only 8-second clips from 180-second videos at each training step, and uses only 40-second content for inference.
We expect \textsc{ClipBERT}'s performance to be further improved by sampling more clips during training and inference.
Meanwhile, we also encourage future work to explore combining extra input signals, such as audio, into the \textsc{ClipBERT}~framework for better performance.

\paragraph{Video Question Answering.}
Table~\ref{tab:main_video_qa} shows evaluation results on video question answering.
Across all three tasks, \textsc{ClipBERT}~achieves significant performance gain.
In Table~\ref{tab:main_tgif_qa}, \textsc{ClipBERT}~1\x1 outperforms prior state-of-the-art QueST~\cite{jiang2020divide} by 6.9\%, 6.8\%, and 0.6\% on TGIF-QA Action, Transition, and FrameQA tasks, respectively.
This is especially surprising considering \textsc{ClipBERT}~1\x1 uses only a single randomly sampled frame from the videos at each training step, while QueST uses 10 uniformly sampled frames.
Moreover, when using only a single frame (the middle frames of the videos) for inference, \textsc{ClipBERT}~1\x1 ($N_{test}{=}1$) already far outperforms QueST on Action and Transition tasks, and is on par with QueST on FrameQA.
In Table~\ref{tab:main_msrvtt_qa}, \textsc{ClipBERT}~4\x1 outperforms HCRN~\cite{le2020hierarchical} by 1.4\% on MSRVTT-QA.
Note that HCRN adopts a sophisticated hierarchical relation modeling network over the entire video sequence of 24 clips at training time, while we use only four randomly sampled frames.
Using more frames further increases this performance gap to 1.8\%.
Table~\ref{tab:main_msrvtt_multiple_choice_test} shows \textsc{ClipBERT}~8\x2 improves ActBERT~\cite{zhu2020actbert} model pre-trained on HowTo100M by 2.5\%, on MSRVTT multiple choice test task.

\section{Conclusion}\label{sec:conclusion}
We present \textsc{ClipBERT}, a generic framework for end-to-end video-and-language learning, which adopts sparse sampling to use only a few sampled short clips from the videos at each training step.
Experiments across diverse tasks show that \textsc{ClipBERT}~outperforms (or is on par with) state-of-the-art methods with densely sampled offline features, suggesting that the ``\textit{less is more}" principle is highly effective in practice.
Comprehensive ablation studies reveal several key factors that lead to this success, including sparse sampling, end-to-end training, and image-text pre-training.

\smallskip
\noindent
{\bf Acknowledgements:} This research was partially done when Jie was an intern with Microsoft. He was later supported at UNC by NSF Award \#1562098, DARPA KAIROS Grant \#FA8750-19-2-1004, ARO-YIP Award \#W911NF-18-1-0336, and Microsoft Investigator Fellowship. The views contained in this article are those of the authors and not of the funding agency.

\section{Additional Experiments}

\paragraph{Visual Question Answering.} As \textsc{ClipBERT}~is designed based on 2D CNN, and is pre-trained on image-text corpus, it is also directly applicable to image-text downstream tasks, such as image-based question answering.
We show \textsc{ClipBERT}'s performance on VQA 2.0 dataset~\cite{goyal2017making} in Table~\ref{tab:main_vqa}.
The model is finetuned from the image-text pre-trained weights on 8GPUs for 13 epochs,  with batch size 32 and learning rate 5e-5.
\textsc{ClipBERT}~shows a reasonable performance compared to the strong pre-training baselines.
Note that \textsc{ClipBERT}~uses grid features~\cite{jiang2020defense,huang2020pixel} instead of the commonly used region features, which is much more computation efficient, \eg, extracting grid features is around 80\x~faster than extracting region features according to the computation time reported in~\cite{jiang2020defense}.

\begin{table}[t]
\tablestyle{3pt}{1.1}
\def19pt{40pt}
\begin{tabular}{l|c|cc}
\multicolumn{1}{c|}{Method} & feature & test-dev & test-std \\
\shline
BUTD~\cite{anderson2018bottom} & R & 65.32 & 65.67\\
grid-feat~\cite{jiang2020defense} & G & 66.47 & -\\
\hline
ViLBERT~\cite{lu2019vilbert} & R & 70.55 & 70.92 \\
VL-BERT~\cite{su2019vl} & R & 71.16 & - \\
Pixel-BERT~\cite{huang2020pixel} & G & 71.35 & 71.42 \\
LXMERT~\cite{tan2019lxmert} & R & 72.42 & 72.54 \\
UNITER~\cite{anderson2018bottom} & R & 72.70 & 72.91\\
Oscar~\cite{li2020oscar} & R & 73.16 & 73.44 \\
\hline
\textsc{ClipBERT}~1\x1 & G & 69.08 & 69.43\\
\end{tabular}
\caption{
\textbf{Comparison with state-of-the-art methods on VQA}.
\textit{G} stands for grid features, \textit{R} stands for region features.
}
\label{tab:main_vqa}
\end{table}

\section{Downstream Task Adaptation}
Our \textsc{ClipBERT}~is quite generic, once trained, it can be easily adopted and transferred for various downstream tasks.
In particular, in this work, we focus on text-to-video retrieval and video question answering.

\paragraph{Text-to-video Retrieval.}
We use a two-layer MLP with the last layer \texttt{[CLS]} token hidden state for a two way (\ie, matched or not matched) classification for retrieval.
We use LogSumExp loss for training. Denote the two-way classification logit output for clip $\tau_i$ from the video associated with the $j$-$th$ example as ${\bm{g}}_{\tau_i}^{(j)} \in \mathbb{R}^2$, where $i=1,\ldots, N_{train}$ for training ($i=1,\ldots, N_{test}$ for inference; see Section 3 of the main paper).
The LogSumExp prediction ${\bm{p}}^{(j)} \in \mathbb{R}^2$ is defined as:
\begin{equation}\label{eq:supp1}
{\bm{p}}^{(j)}=\frac{\sum_{i=1}^{N_{train}}{e^{{\bm{g}}_{\tau_i}^{(j)}}}}{\text{sum}(\sum_{i=1}^{N_{train}}{e^{{\bm{g}}_{\tau_i}^{(j)}}})}.
\end{equation}

\noindent
We then use a negative log likelihood loss for training:
\begin{equation}\label{eq:supp2}
L = - \frac{1}{|\mathcal{D}|} \sum_{j=1}^{|\mathcal{D}|}\mathrm{log}{\bm{p}}^{(j)}[y_j],
\end{equation}
where $\mathcal{D}$ is the dataset, $y_j$ is the index of the ground-truth answer for the $j$-$th$ example.

We conduct experiments on three popular text-to-video retrieval datasets, MSRVTT~\cite{xu2016msr}, DiDeMo~\cite{anne2017localizing}, and ActivityNet Captions~\cite{krishna2017dense}.
Table~\ref{tab:training_details_text_to_video} shows the training details for models on each of the datasets.

\begin{table}[h]
\tablestyle{4pt}{1.1}
\def19pt{20pt}
\scalebox{1.0}{
\begin{tabular}{l|ccc}
Dataset & \#Epochs & Bsz \x Grad-Accu \x \#GPUs  & LR \\
\shline
MSRVTT & 20 & 16\x1\x8 & 5e-5 \\
DiDeMo & 20 & 8\x4\x8 & 5e-5\\
ActivityNet Captions & 80 & 16\x2\x8 & 5e-5\\
\end{tabular}
}
\caption{
\textbf{Training details for text-to-video retrieval tasks}. Bsz is short for batch size. Grad-Accu stands for gradient accumulation steps. LR means initial learning rate.
}
\label{tab:training_details_text_to_video}
\end{table}

\paragraph{Video Question Answering.}
Similar to text-to-video retrieval task, we take the last layer \texttt{[CLS]} token hidden state through a two-layer MLP for classification.
We use LogSumExp to aggregate prediction from multiple clips to calculate loss. The formulation of LogSumExp loss is simlar to Equation~\ref{eq:supp1} except that the dimension of ${\bm{g}}_{\tau_i}$ equals to the number of answer candidates.

We conduct experiments on three video QA datasets, TGIF-QA~\cite{jang2017tgif}, MSRVTT-QA~\cite{xu2017video}, and MSRVTT MC Test~\cite{yu2018joint}.
For TGIF-QA, we evaluate three sub-tasks, \ie, Action, Transition, and FrameQA. We train a separate model for each of the evaluated TGIF-QA tasks.
For MSRVTT MC Test, as it uses the same training set as the MSRVTT retrieval task, we directly use the trained retrieval model to rank the five candidate answers.
Table~\ref{tab:training_details_text_to_video} shows the training details for models on TGIF-QA tasks and MSRVTT-QA.

\begin{table}[h]
\tablestyle{4pt}{1.1}
\def19pt{20pt}
\scalebox{1.0}{
\begin{tabular}{l|ccc}
Dataset & \#Epochs & Bsz\x Grad-Accu \x \#GPUs & LR \\
\shline
TGIF-QA Action & 55 & 32\x1\x8 & 1e-4 \\
TGIF-QA Transition & 15 & 32\x1\x8 & 1e-4 \\
TGIF-QA FrameQA & 15 & 32\x1\x8 & 1e-4 \\
MSRVTT-QA & 10 & 16\x1\x4 & 5e-5 \\
\end{tabular}
}
\caption{
\textbf{Training details for video question answering tasks}. Bsz is short for batch size.
Grad-Accu stands for gradient accumulation steps.
LR means initial learning rate.
}
\label{tab:training_details_video_qa}
\end{table}

redpajama_set_name: RedPajamaArXiv


text:
const mockObjects = [{
  id: 'abc-news-au',
  name: 'ABC News (AU)',
  description: `Australia's most trusted source of local,
   national and world news.
  Comprehensive, independent, in-depth analysis,
  the latest business, sport, weather and more.`,
  url: 'http://www.abc.net.au/news',
  category: 'general',
  language: 'en',
  country: 'au',
  urlsToLogos: {
    small: '',
    medium: '',
    large: ''
  },
  sortBys: [
    top
  ]
},
{
  id: 'ars-technica',
  name: 'Ars Technica',
  description: `The PC enthusiast's resource.
  Power users and the tools they love, without computing religion.`,
  url: 'http://arstechnica.com',
  category: 'technology',
  language: 'en',
  country: 'us',
  urlsToLogos: {
    small: '',
    medium: '',
    large: ''
  },
  sortBys: [
    'top',
    'latest'
  ]
}];

export default mockObjects;

redpajama_set_name: RedPajamaGithub


text:
Fans Think Axel Tuanzebe Was Wearing An Apple Watch While Playing For Manchester United
Published 19:30, 11 May 2021 BST
| Last updated 22:11, 11 May 2021 BST
Axel Tuanzebe hit headlines tonight, with fans thinking he had his Apple watch on during tonight's game against Leicester City.
The academy product was handed a start at Old Trafford this evening as Ole Gunnar Solskjaer made ten changes in preparation for the big match against Liverpool in 48 hours.
And while partnering Eric Bailly at the heart of United's backline, the 23-year-old had his watch on his wrist - clearly trying to get those extra steps in for the day.
Fans had never seen anything like this before in the Premier League and so there was a wave of reaction on social media:
Tuanzebe's watch wasn't actually an apple one, it was a whoop fitness tracker watch that is used by NFL players and professional golfers.
If Tuanzebe had alerts from the Premier League app on his watch, then it would have told him that his side's 2-1 defeat at home to Leicester means rivals Manchester City have now won the title for the fifth time.
City, who are set to receive a guard of honour against Newcastle on Friday, have now finished above United for eight straight seasons.
Topics: Axel Tuanzebe, Leicester City, Football, Manchester United
Manchester United fans think they've found the perfect solution for Casemiro's suspension
Newcastle United fans certain club are signing James Maddison after seeing his social media
Arsenal fans in Uganda arrested for celebrating Man United win questioned as they 'didn't know when Invincibles was'
Manchester United fans point out substitution is to blame for conceding late goals

redpajama_set_name: RedPajamaCommonCrawl


* - Denotes a Quick Ship Leather or Fabric Color Chair.
** - Denotes a Custom Order 3-4 month lead time color or fabric.
This Stressless Recliner features a forward adjustable headrest. The headrest is not height adjustable.


Adams, John: "Plaintiff was Adams, Defendant was Alzail Shearman, Assumpset.
Witnesses were Jesper Hester and Elijah Fenn."
The deed is recorded in grant book N-5, page 275.
Fenn, Elijah: "Appointed to view road from Dublin to road that intersects the road in Montgomery at Ephraim Turners."
"B. ELIJAH FANN (Notice he used the FANN spelling) was born on 17 July 1788 and died 5 Jan 1844 in Early (now Miller) Co, GA. He married MARTHA RICH in 1807 in Burke Co, GA according to Martha's 1855 application for a land grant. MARTHA RICH was born on 3 Sept 1795 and died in Miller County on 10 May 1859. Travis Fenn, believed to be the father of Elijah, was living in Burke County in 1807. Travis has a court record dated 9 Jan 1806 in Burke County.
Dot Arthur, POB 1635, Panama City, FL 32402 has copied information from her great grandmother's family Bible giving Elijah & Mary's dates of birth and death, and Joe T. Smith, 423 Meadowlark Dr., Albany, GA 31707 has done extensive research on Elijah & Martha (Rich) Fann.
Neither knows who Elijah's father is, but suspect Travis Fenn may be. Elijah is named as a possible son of Travis Fenn.
(NOTE: He was actually about 25 when he enlisted.) He served in the same military unit during the War of 1812 as Zacheus Fann, son of Travis Fann. John Fann, Jr. was also in the same unit. John Fann Jr. was a son of John Fann Sr., a brother of Zachariah. They served as Privates in Capt. Thomas Company of Infantry, Wimberly's Brigade, Georgia Militia. The three Fanns were discharged at Darrien, GA and paid $32.52 ($8 per month pay + $.16 per day for food for 4 days). They left for their homes in Dublin, Laurens Co., GA, a distance of about 200 miles.
The deaths of Elijah and Martha Fenn are recorded in the Bibble of Leonidas Duke Sr and his wife Melvina Fann."
Fenn, Elijah: "Selected as Inferior Court juror for next term of court."
the same from the claims of all other persons.
In witness I set my hand and seal this day and date above written.
12 Jno H Gray (Folbs)"
The Early County court in its Jul 1837 term, "appointed Elijah Fenn, Moses Warren, Delaware Freeman, Council B. Allen, Elijah Fenn and John R. Miller" to a committee to find a site on which to build a bridge across Spring Creek.
They "reported in favor of a place on lot 241 in the 13th district, belonging to Moses Warren. This report was accompanied by a petition from a number of citizens of the county asking for the adoption of the report of the commissioners. Whereupon the court ordered the bridge to be built at the point recommended and instructed the commissioners to let out the building of the same, which they proceeded to do. It was bid off by James G. Collier at the price of $465. This is the bridge that is now on the road between Charles Roberts' and the late Henry Batts'--or rather it was built at the same place. The first bridge was built in January, 1838."
This creek flows north to south through the area which is present-day Miller County and is wide, with low, shallow banks and a relatively broad, dense hardwood edge which runs back for several hundred yards. The committee shortly recommended that the bridge be built not too far south of the present U. S. Highway bridge at Colquitt. There is still a bridge for local use where the committee recommended.
This is the part of Early County from which Miller County was formed in 1856.
This folded lettersheet / stampless letter has a circular date stamp for IRWINTON Ala. NOV, [$30 catalog value, American Stampless Cover Catalog, 1985], a 12 1/2 cent rate, and is addressed to Mr. Elijah Fenn, Bainbridge, Georgia, and is a one paragraph message written by Elijah Fenn. The headline is September the 5th, 1842.
The letter reads: "I give to Adaline Griffin thirteen head of stock Cattle and mark and brand mark crop and half cross in the left year [sic] and two under betts [?] in the side Recorded by me and brand A G."
Apparently, Fenn was out of town, and so addressed this letter to himself, and the Post Master at the General Store (or such) in Bainbridge had probably been told to open his mail in Fenn's absence, and the Post Master took this to the courthouse, where the county clerk signed noted it "recorded" and signed it "A.D. Smart, Clk." Decatur county.
It's a quarter of a full lettersheet. Maybe there was a shortage of paper, or there was other writing that was torn off and long gone, who knows. Postal marking cover is all there.
Q: Hi! Are you sure the letter is dated September, as the postmark is clearly November?
A: Hello and thanks for the question. It's good observation that you make. The handwritten date by Fenn on the inside of the letter is September the 5th, 1842. The date written by the Decatur GA clerk is Sept 16th, 1842. What I'm thinking -- this may have been a turned cover. That is, it was a stampless letter mailed in September to Irwinton, ALA to an unknown person. That person then tore off the side of the lettersheet that was addressed to him, likely noted it's contents, perhaps even had an extra enclosure to mail back, now long gone, then remailed the same sheet back to Fenn in Georgia, from Irwinton in November. There were lots of so called turned covers in those days. That's my best guess. And so it's likely that Fenn himself wasn't in Irwinton.
I do hereby certify that the above appraisers were sworn to perform their duty as appraisers according to law this the 23rd of August 1844 Collen Cowart J.P.
William Grimes, administrator for the estate of Elijah Fenn, posted the following notice in the Albany, Georgia newspaper: Georgia, Early County: Will be sold before the courthouse door in said county on the first Tuesday in August, next, between the legal hours of sale, all the property belonging to the estate of Elijah Fenn, late of said county, deceased. Sold for the benefit of the heirs and creditors of said estate. (Signed) William Grimes, Adm.
William Grimes, administrator for the estate of Elijah Fenn, posted the following notice in the Albany, Georgia newspaper: Georgia, Early County: Will be sold before the Court House door in said county on the first Tuesday in October, next, between the legal hours of sale, all the property belonging to the estate of Elijah Fenn, late of said county, deceased. Sold for the benefit of the heirs and creditors of said estate. (Signed) William Grimes, Adm.
hundred dollars, Payable by the said Leonidus P.
He was buried at Mothers Home Primitive Baptist Church cemetery, which was founded in 1922. Pilgrims Rest-Whites Bridge Church and cemetery are a few miles South on County Line Road.
[S1228] Foot Prints & Wagon Tracks, 2nd ed., Doyle Fenn, (Manvel, Texas: published by author, 2000), 28-31, 5 Jan 2008.
[S19] Personal Correspondence, e-mail from Jack Carroll Cumbie of Ozark, Alabama [Jack43188@aol.com], a descendant of Eliza Fenn Hatten, to Gail Rich Nestor dated 26 September 2006, 3 Jan 2008.
[S19] Personal Correspondence, e-mail from Tim Griffin of Fort Lauderdale, Florida [timg_2001@yahoo.com], a descendant of Sarah Fenn Griffin, to Gail Rich Nestor dated 18 August 2005, 5 Jan 2008.
[S19] Personal Correspondence, e-mail from Marvin Reed of New York, New York [tworeeds1@verizon.net], a descendant of Sarah Fenn Griffin, to Gail Rich Nestor on 15 June 2007, 5 Jan 2008.
[S19] Personal Correspondence, e-mail from Betty Fann Gallagher [yelowros36@hotmail.com], a descendant of William Fenn, to Gail Rich Nestor dated 2 January 2008, 5 Jan 2008.
[S983] Montgomery County, Georgia, A Source Book of Genealogy and History, James E. Dorsey and John K. Derden, (Spartanburg, South Carolina: The Reprint Company Publishers, 1983), G975.8964 DAV GA., 102.
[S1192] Montgomery County, Georgia, "Deed Book H, 1809-1817," on microfilm in drawer 137, box 48, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), 175-6, 26 Jun 2007.
[S1192] Montgomery County, Georgia, "Deed Book H, 1809-1817," on microfilm in drawer 137, box 48, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), 177-8, 26 Jun 2007.
[S617] Unbound Headright and Bounty Documents Files, unit 95-2353A in private viewing area, Georgia Surveyor General Department, (Morrow, Georgia: Georgia Archives), Elijah Fenn grant, 1814, box 8055-03b, file 78.
[S969] Index to War of 1812 Service Records for Volunteer Soldiers From Georgia, Judy Swaim Kratovil, abstracter, (Atlanta, Georgia: J.S. Kratovil, 1986), 46.
[S771] Laurens County, Georgia Legal Records, 2 volumes, Allen Thomas, (Roswell, Georgia: W. H. Wolfe Associates, 1991), G929.3758 WILKINS., 1:13; citing key 49, page 21, 27 Jun 2007.
[S872] The Third or 1820 Land Lottery of Georgia, Rev. Silas Emmett Lucas, Jr., (Easley, South Carolina: Southern Historical Press, 1986), G 929.3758 GEO GEORGIA., 107.
[S919] Index to the Headright and Bounty Grants of Georgia, 1756-1909, Silas Emmett Lucas, (Vidalia, Georgia: Georgia Genealogical Reprints, 1970), 195.
[S852] "Laurens County, Georgia Land Grants, 1784-1903," database, Laurens County, Georgia Rootsweb, (http://ftp.rootsweb.com/pub/usgenweb/ga/laurens/deeds/grants.txt), grant to Elijah Fenn, 1821.
[S771] Laurens County, Georgia Legal Records, 2 volumes, Allen Thomas, (Roswell, Georgia: W. H. Wolfe Associates, 1991), G929.3758 WILKINS., 1:240; citing key 3, page 113, 27 Jun 2007.
[S771] Laurens County, Georgia Legal Records, 2 volumes, Allen Thomas, (Roswell, Georgia: W. H. Wolfe Associates, 1991), G929.3758 WILKINS., 1:240; citing key 3, page 136, 27 Jun 2007.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elisha Fenn entry, Captain Douglas' District, 1825 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elisha Fenn entry, Captain Douglas' District, 1826 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fann entry, Douglas' District, tax year 1827.
[S951] Decatur County, Georgia, Minute Books, Clerk of Court, (Probate Court, Bainbridge, Georgia), 19.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fann entry, Captain McElves' District, 1828 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elisha Fenn entry, Captain Hilling's District, 1829 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fenn entry, Captain Wester's District, tax year 1830.
[S531] Decatur County, Georgia, General Index to Realty Deeds & Mortgages, Clerk of Court, (Superior Court, Bainbridge, Georgia), grantee index 5A, 27 Apr 2007.
[S877] Decatur County, Georgia, Real Estate Conveyance Books, Clerk of Superior Court, (Superior Court, Bainbridge, Georgia), B:44, 27 Apr 2007.
[S877] Decatur County, Georgia, Real Estate Conveyance Books, Clerk of Superior Court, (Superior Court, Bainbridge, Georgia), M:327, 28 Apr 2007.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elisha Fenn entry, Captain Westers' District, 1831 tax year.
[S951] Decatur County, Georgia, Minute Books, Clerk of Court, (Probate Court, Bainbridge, Georgia), unlettered book, 1824-1847, page 92, 7 Mar 2007.
[S777] "Georgia Cherokee Land Lottery, 1832," subscription database, Ancestry.com, (www.ancestry.com), Elijah Fenn, fortunate drawer no. 23.
[S874] The 1832 Gold Lottery of Georgia: Containing a List of the Fortunate Drawers in Said Lottery, Silas Emmett Lucas, (Easley, South Carolina: Southern Historical Press, 1988), G 929.3758 GEO GEORGIA., 161.
[S951] Decatur County, Georgia, Minute Books, Clerk of Court, (Probate Court, Bainbridge, Georgia), unlettered book, 1824-1847, page 96, 7 Mar 2007.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fenn entry, Captain Caswell's District, 1833 tax year.
[S951] Decatur County, Georgia, Minute Books, Clerk of Court, (Probate Court, Bainbridge, Georgia), unlettered book, 1824-1847, page 113, 7 Mar 2007.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fenn entry, Captain Caswell's District, 1834 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elijah Fenn entry, Captain Williams' District, 1835 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elija Fan entry, Captain Smart's District, 1838 tax year.
[S786] Decatur County, Georgia, "Various Tax Lists, 1824-1850," on microfilm in drawer 127, box 10, Clerk of Superior Court, (Morrow, Georgia: The Georgia Archives), Elija Fan entry, Captain Smart's District, 1839 tax year.
[S531] Decatur County, Georgia, General Index to Realty Deeds & Mortgages, Clerk of Court, (Superior Court, Bainbridge, Georgia), grantor index 5A, 27 Apr 2007.
[S877] Decatur County, Georgia, Real Estate Conveyance Books, Clerk of Superior Court, (Superior Court, Bainbridge, Georgia), E:9, 29 Apr 2007.
[S1719] 1840 U.S. census, Ancestry.com, Digital images, Georgia, Early County, (http://www.ancestry.com: National Archives and Records Administration, 2004), M704, roll 40, Wilson Township, p. 115, Elijah Fann household, accessed 11 Jan 2011., 11 Jan 2011.
[S615] Early County, Georgia, "1842 Tax Digest," on microfilm in drawer 32, box 2, (Morrow, Georgia: The Georgia Archives), Elijah Fenn tax entry.
[S877] Decatur County, Georgia, Real Estate Conveyance Books, Clerk of Superior Court, (Superior Court, Bainbridge, Georgia), F:103, 29 Apr 2007.
[S788] Early County, Georgia, "Inferior Court Minutes, Book A: 1820-1849," on microfilm in drawer 32, box 13, (Morrow, Georgia: The Georgia Archives), Letters of Administration for estate of Elijah Fenn, July Term 1844 [un-numbered page].
[S614] Early County, Georgia, "Estate Records-Inventories, Appraisements & Sales, Book B: 1839-1852," on microfilm in drawer 31, box 65, (Morrow, Georgia: The Georgia Archives), inventory and appraisement of the estate of Elijah Fenn, 106.
[S614] Early County, Georgia, "Estate Records-Inventories, Appraisements & Sales, Book B: 1839-1852," on microfilm in drawer 31, box 65, (Morrow, Georgia: The Georgia Archives), sale of estate of Elijah Fenn, 106-107.
[S788] Early County, Georgia, "Inferior Court Minutes, Book A: 1820-1849," on microfilm in drawer 32, box 13, (Morrow, Georgia: The Georgia Archives), Appraisement & Sale filing for the estate of Elijah Fenn, November Term 1844 [un-numbered page].
[S788] Early County, Georgia, "Inferior Court Minutes, Book A: 1820-1849," on microfilm in drawer 32, box 13, (Morrow, Georgia: The Georgia Archives), Leave to sell application for estate of Elijah Fenn, May Term 1845 [un-numbered page].
[S765] Albany, Georgia, Newspaper Clippings, 3 volumes, Tad Evans, (Savannah, Georgia: Tad Evans, 1996), G929.3758 BAK GA., 1:6.
[S765] Albany, Georgia, Newspaper Clippings, 3 volumes, Tad Evans, (Savannah, Georgia: Tad Evans, 1996), G929.3758 BAK GA., 1:13.
[S779] Early County, Georgia, "Early County Guardians and Administrators Bonds, 1834-1902," on microfilm in drawer 31, box 66, (Morrow, Georgia: The Georgia Archives), guardianship bond for orphans of Elijah Fenn, p. 142.
[S779] Early County, Georgia, "Early County Guardians and Administrators Bonds, 1834-1902," on microfilm in drawer 31, box 66, (Morrow, Georgia: The Georgia Archives), guardianship bond for orphans of Elijah Fenn, p. 142-3.