We propose fill-in-the-blanks as a video understanding evaluation framework and introduce FIBER  a novel dataset consisting of 28,000 videos and descriptions in support of this evaluation framework. The fill-in-the-blanks setting tests a model's understanding of a video by requiring it to predict a masked noun phrase in the caption of the video, given the video and the surrounding text. The FIBER benchmark does not share the weaknesses of the current state-of-the-art language-informed video understanding tasks, namely: (1) video question answering using multiple-choice questions, where models perform relatively well because they exploit linguistic biases in the task formulation, thus making our framework challenging for the current state-of-the-art systems to solve; and (2) video captioning, which relies on an open-ended evaluation framework that is often inaccurate because system answers may be perceived as incorrect if they differ in form from the ground truth. The FIBER dataset and our code are available at https: //lit.eecs.umich.edu/fiber/ . Despite current progress on multimodal (textual and visual) representations, language-informed video understanding is still a very challenging task for machine learning systems (Zhang et al., 2021; Li et al., 2021). This is due in large part to the task setup and the dataset construction. Current video understanding datasets often have at least one of two major limitations. First, they have limited application value. E.g., multiple-choice questions (Lei et al., 2018; Tapaswi et al., 2016; Jang et al., 2017; Castro et al., 2020) do not reflect real-world tasks. Second, they are based on subjective evaluation metrics, e.g., video captioning (Tran et al., 2016; Krishna et al., 2017; Zhou et al., 2018; Wang et al., 2019)), and are therefore hard to evaluate automatically, as the ground truth can be expressed in different ways. In this paper, we address these limitations by introducing a new dataset named FIBER that collects multiple perspectives on the same video, focusing on noun phrases as a proxy for different entities and their interactions in the video. Our data focuses on recall and tests the ability of models to capture a wide range of possible interpretations for a particular aspect of a video. We construct the FIBER dataset by systematically blanking captions from an existing video captioning dataset named VaTeX (Wang et al., 2019) and by providing additional correct answers for the blanks. VaTeX is a video captioning dataset that contains 40,000 10-second YouTube videos with 10 English captions per video. 1 We build our 1 Licensed under Creative Commons, more information here: https://eric-xw.github.io/ vatex-website/index.html . video fill-in-the-blanks dataset by blanking random noun phrases from one of the English captions for each video, from a subset of VaTeX consisting of 28,000 videos. Through extensive analyses, we show that the blanked noun phrases are essential for understanding important visual aspects from the video. To address the fill-in-the-blanks task, we propose a Transformer-based (Vaswani et al., 2017) multimodal model. Our experiments show that our best multimodal model achieves a token-level F1 score of 71.4 while the F1 score of crowd workers is 82.5, indicating that this task is challenging for video and text understanding. The contribution of this work is threefold: (1) We propose a novel fill-in-the-blanks task as an evaluation framework that addresses the drawbacks associated with previous approaches to video understanding. In support of this framework, we introduce FIBER , which is a novel dataset of 28,000 videos and fill-in-the-blanks captions with multiple correct answers. (2) We propose several unimodal baselines and two multimodal models for solving this task. (3) We provide a detailed analysis of the data to measure the diversity and complexity of the answers, and also conduct an error analysis of the models' performance, to gain insights into the blanked captions and videos that are hard for the models to solve. Language-informed video understanding is a complex task that has been extensively addressed in the multimodal (natural language and computer vision) machine learning research through diverse tasks and benchmarks. Multiple-Choice Video Understanding. Multiple-choice benchmarks consist of identifying the only correct answer from a set of distractors, where the set of possible answers varies depending on the input. Video Question Answering (Video QA), a popular format, consists of answering questions based on the video content. Numerous multiple-choice Video Understanding benchmarks have been proposed such as TVQA (Lei et al., 2018), MovieQA (Tapaswi et al., 2016), TGIF-QA (Jang et al., 2017) (Repetition Action and State Transition tasks), LifeQA (Castro et al., 2020), PororoQA (Kim et al., 2017), Mari-oQA (Mun et al., 2017), VCQA (Zhu et al., 2017), VideoMCC (Tran et al., 2016), and ActivityNet QA (Yu et al., 2019). However, they provide choices and are thus easier to solve than generating arbitrary text. A further drawback is that the performance without the visual input is generally already high as models are able to exploit biases in the dataset (Agrawal et al., 2018) or they count on other modalities that overlap in functionality with the visual one. Video Captioning. Video Captioning consists of generating a piece of text that describes a given video. This task can be carried out using multiple datasets such as ActivityNet Captions (Krishna et al., 2017) (also features Dense-Captioning), YFCC100M (Thomee et al., 2016), (Alayrac et al., 2016), DiDeMo (Anne Hendricks et al., 2017), MSR-VTT (Xu et al., 2016), YouCook2 (Zhou et al., 2018), How2 (Sanabria et al., 2018), HowTo100M (Miech et al., 2019), VaTeX (Wang et al., 2019), TGIF (Li et al., 2016), MovieNet (Huang et al., 2020), LSMDC (Rohrbach et al., 2017), TGIF-QA (Li et al., 2016) (Frame QA task). Due to the diversity of captions provided, Video Captioning benchmarks do not present a high human agreement and are thus hard to evaluate automatically with certainty (Aafaq et al., 2019). Video Understanding Based on Filling Blanks. VideoBERT (Sun et al., 2019b), CBT (Sun et al., 2019a), UniVL (Luo et al., 2020), ActBERT (Zhu and Yang, 2020), and HERO (Li et al., 2020) methods propose masking random parts of the input from text and video pairs for training. However, they do this only for the purpose of system training and do not use the framework to test and evaluate video understanding. The only exception is MovieFIB (Maharaj et al., 2017) which employs a video fill-in-the-blanks scheme, based on LSMDC (Rohrbach et al., 2017) for both training and evaluation. However, these methods have several drawbacks. They blank a single word, which makes it easier to guess; they evaluate correctness with a single ground-truth answer per caption; and they focus on the movies domain (we focus on YouTube videos). Concurrent Work. The most similar work to ours is VidQAP (Sadhu et al., 2021), which presents an evaluation framework to fill in blanks with phrases using semantic roles based on ActivityNet Captions (Krishna et al., 2017) and Charades (Sigurdsson et al., 2016); unlike this existing work, we design our benchmark to feature a high 2926 human accuracy (avoiding ActivityNet Captions as it is contextualized, collecting multiple correct answers, and showing a high human performance). Our work is also close to (Yang et al., 2021) on evaluating the use of free-form QA; however, they employ a small vocabulary and no human accuracy that serves as an upper bound for the task. The novelty of our work lies in our use of a hard task (a considerable gap between human and best model performance) that measures a form of video understanding while at the same time yielding a high human performance due to the large number of possible correct answers we collected (  13 per caption) from multiple annotators (  9 per caption). We construct FIBER  a large video understanding dataset that can evaluate the ability of a model to interpret and use a multimodal context by requiring the models to fill in (generate) a blank (a missing constituent) in this context. We build FIBER by following two main steps: (1) data generation, where we compile a large set of video-caption pairs with selectively blanked words; and (2) data annotation, where crowd workers provide additional valid answers for these blanks. Note that we could also develop a fill-in-the-blanks dataset by completing only the first step: the data generation. However, this would result in only one valid answer (the original blanked word or phrase), which can lead to unfair evaluations that are too strict because of alternative correct answers being dismissed (e.g., child provided as an answer where the blanked word was kid). Other than manual annotations, we found no high-quality method to automatically obtain additional correct answers. For example, building and t-shirt in Table 7 are too dissimilar but both are correct, pink and yellow in Fig. 1 are semantically close but only one is correct. The dataset is constructed starting with the VaTeX (Wang et al., 2019) dataset. VaTeX is a multilingual video captioning dataset, consisting of over 41,250 video clips, each of which is taken from a unique public YouTube video, and lasts around 10 seconds. For each video clip, there are 10 English and 10 Chinese captions associated with it. to mask only noun phrases for three main reasons. First, noun phrases often require visual information for identification or understanding. They cover a large variety of information regarding visual content, as their head nouns can describe people, objects, scenes, events, and more. A model often needs to identify the related objects in the videos, as well as the properties of objects (e.g., color, number, or size) to fill the blank correctly. Second, nouns are usually essential to understanding of visual content and serve as reliable predictors of the ability of a system to understand a video. Other phrases, such as verbs or adjectives, can more easily be guessed from the text only while ignoring the visual information. To illustrate, consider the example A woman _____ in the pool, where a model can easily predict that the blank should be swims from the textual content only, which would not be the case for A woman swims in _____, where the blank could be completed by sea, pool, lake, water, and other similar nouns. Third, in preliminary experiments, we found that nouns lead to more robust annotations as compared to e.g., adjectives, which can have low inter-annotator agreement due to their subjectivity. As an example, consider the phrase A _____ hill stands behind the house. where the blank could be filled with a color property, a size property, or another attribute. For each video, we choose the first English caption that contains at least one noun phrase as detected by spaCy 2 (Honnibal et al., 2020), and randomly blank one of these noun phrases to generate an instance. Accordingly, we generate our training, validation, and test data starting with the VaTeX v1.1 training set, a random subset of size 1,000 from the validation set, and a random subset of size 1,000 from the test set, respectively. We performed a crowdsourced annotation procedure to collect additional correct answers for each blank in the validation and test sets. As highlighted earlier, the main reason for collecting these additional annotations is to reflect the natural diversity of language, and have multiple alternative answers for each blank. We use Amazon Mechanical Turk (AMT) for the annotation. Figure 2 shows the annotation interface 2 We used the model en_core_web_trf from spaCy v3. An error analysis identified only three tagging errors in a sample of 247 sentences. and a highlight of the data collection instructions (additional guidelines were provided, not shown here for space reasons). For each blanked caption, workers were presented a video clip along with the corresponding masked caption. They were then asked to fill in the blank with a noun phrase. 3 We also asked annotators to provide answers in a confidence-descending order (the first answer should be the most natural one to the annotator). We presented five videos in each Human Intelligence Task (HIT). Nine workers annotated each of them with at least two answers for each blank. We paid a bonus for each extra answer for each blanked caption, from the second one to the fifth one, to encourage them to provide more answers. We calculated a $12 hourly rate for a worker that provides at least five answers. We estimated the time to annotate one video to be 30 seconds. Consequently, the HIT pay rate was $0.2, which could result in a total of $0.5 with the added bonus. Additionally, we offered another type of bonus of $0.2 to the worker with the largest number of correct answers for every HIT, to encourage them to provide more than five answers. We required workers to be in Canada or the United States, 4 and to have completed at least 1,000 3 We blanked multi-word spans for the task, rather than single-word noun phrases, because blanking a single noun at a time led to a lower annotator agreement in preliminary experiments, likely due to the lower likelihood of overlap. For example, annotator 1 might write young boy and annotator 2 might write young child, which would have at least some overlap as compared to boy and child (no overlap). 4 We restricted the task to these countries because it is a good proxy for proficient English speakers and because our task received lower-quality responses otherwise. HITs on AMT with at least a 92% approval rate. The interface also checked that for a given worker and caption the answers were different. For this, we first normalized the answers by lower-casing, stripping punctuation and extra spaces, and removing the determiners the, a, and an. During the annotation, we manually reviewed a sample to identify cases of incorrectly tagged noun phrases (e.g., inside marked as a noun when it should be a preposition) and factually incorrect noun phrases (e.g., referring to bags as eggs without any information on the contents of the bags); we disqualified workers who consistently provided incorrect annotations. After collecting annotations, we filtered for noun phrases using the same method as before, based on whether the text is parsed as a noun phrase (including bare nouns, e.g. man is walking), a wh-phrase (who is speaking), a simple gerund (eating is a good way to stay healthy), or infinitive (to eat is wonderful). We compute summary statistics on the annotated data to determine the degree of similarity with the originally blanked phrases. The statistics are shown in Table 1. We find that, in general, annotators tend to provide  3 unique answers for the provided data. Compared to the original phrases, annotators tend to use about the same number of tokens. Annotators also use visual words at a much lower rate than the original phrases, possibly because the task encouraged the annotators to generate as many distinct nouns as possible without regard to descriptive information. To further validate the utility of the annotations collected in this study, we provide an extensive analysis of the answers (which is obtained from the union of the annotations and the originally blanked phrases). We compute the most-frequent answers and find, as expected, that noun phrases related to person are the most frequent: the word man appears in 5.7% of total original phrases and 1.2% of total annotations (see Figure 5 in the Appendix). Note that our annotations have a long tail distribution, as the most-frequent noun phrase appears in only 1.2% of total annotations. In addition, we find that answers related to person, such as another person are not trivial. On the contrary, in the third example in Fig. 1, for example, a model has to reason about the actions of both persons and distinguish between them. The other two examples in Fig. 1 also reflect how a model needs to understand both the video and the text in order to complete the blanks. Figure 3 shows what kind of answers are depicted in the videos. This analysis shows the diversity and complexity of answers that a model needs to fill in, demonstrating a strong video understanding. As expected, the cluster Person-related has the most answers, followed by the clusters: Objects (e.g., shoes, glasses), Places (e.g., mountain, street), Materials (e.g., metal, wood), and Body parts (e.g., fingers, head). Note also that the Person-related cluster, among more typical answers such as male and female, also contains complex and diverse answers such as dancer, workers, musician or audience. To establish a reference for the machine models, we compute the agreement among annotators using the evaluation metrics described in Section 5.1, which we also use for model evaluation (Section 5.2). Specifically, we apply a leave-one-out strategy to construct the test set and the ground truth set. We compare the first answer provided by each crowd worker (which is their most natu-ral/confident answer) against the complete set of answers provided by the other crowd workers, using maximum F1 score (token overlap) and maximum exact match (EM) as agreement metrics, as described in Section 5.1. Table 2 shows the inter-annotator agreement. We show the mean values of the agreement metrics per-caption and per-answer (recall there are multiple answers per caption, so in the former case we first average among the answers within the caption and person man guy someone kid woman girl boy child people lady male adult group friend human female athlete he teenager gentleman worker individual young man crowd dude student they teen player she audience young boy toddler another person somebody team baby youngster instructor teacher young girl some dude little girl artist young woman performer group of people trainer young child two people couple employee young lady room video ground building house floor field gym it object camera tool something instrument item food animal machine water board ball thing container chair device equipment toy table stick music lot bunch song piece line other gamecompetition hand wood wall grass dirt front back side Person-relatedPlace ObjectOther Body part Material Position Figure 3: The 2D t-SNE (Van der Maaten and Hinton, 2008) representation of the clustering of the top 100 most frequent answers provided for the blanks. then across the captions). The higher rates of agreement at the caption level, compared to the answer level, indicate a high amount of answer diversity among the workers. To validate the quality of the crowdsourced annotations, we also compare them against human annotations collected from two trusted annotators (both researchers at the University of Michigan). We sample 200 captions from the validation set and ask these two annotators to perform the same labeling task that the MTurk workers performed, and then compare their agreement with the crowdsourced data. The annotators obtain a per-caption average of 90.2% F1 score and 49.0% exact match accuracy, comparable to the agreement scores of the workers. NPs vs. other phrases. By looking at a video and filling a blank caption with a noun phrase can sometimes indirectly capture other aspects such as actions (verbs, adverbs) and object quality (adjec-tives, modifiers). However, this is not always the case. This is especially true for noun phrases that are easier to guess (cf. Table 4). Focus on human actions. Our data focuses mostly on human-related activities (e.g., sports), and may lack general representation available in other datasets related to animals, nature, and technology, to name a few. Availability of the videos. As we build upon VaTeX (Wang et al., 2019) and YouTube, some videos may become unavailable over time. To mitigate this issue, the VaTeX website offers to download pre-extracted video features. 5 Efficiency of the data annotation process. Not all videos have multiple possible captions for noun phrases. For example, the fork may be the only reasonable answer for a given video and blanked caption, and annotators may not have anything else to add. We propose an encoder-decoder multimodal method to perform the task of video fill-in-the-blanks. We first encode the text and visual modalities together to obtain a semantic representation of the blanked caption and video. The decoder uses the semantic representation to generate text corresponding only to the answer to the blank. To correctly generate an answer, a model needs to learn which parts of videos relate to the missing parts of the caption. To accomplish this, we use the original Transformer architecture (Vaswani et al., 2017), whose self-attention mechanism is particularly effective for encoding relations within an input sequence and have been shown to perform well in many language understanding tasks. We consider two types of encoders, namely the early-fusion encoder and the late-fusion (two-stream) encoder. The structure of our multimodal 5 https://eric-xw.github.io/ vatex-website/download.html Transformer Encoder _____ performs a shot put at an outdoor course. model with an early-fusion encoder is shown in Fig. 4a. The input to the model consists of the tok-enized blanked caption-text t 1 , . . . , t n , as well as a representation of the video consisting of multiple video sequence features v 1 , . . . , v m from a video feature extractor. The blanked captions are embedded by an embedding layer. The video features are projected into the encoder by a linear layer. We use a special token to represent the masked phrase and another one to separate the input text and video sequences. We add positional embeddings to each input token or video feature to represent the sequence order, and another embedding to indicate whether it belongs to the text or video sequence similarly to BERT (Devlin et al., 2019). The late-fusion model is shown in Fig. 4b. The late-fusion model encodes the language and video first separately and then jointly. This is because the modalities may benefit from learning independently about their own context before using them together. For the video encoder, we use the existing I3D (Car-reira and Zisserman, 2017) features (size 1024 every 8 consecutive frames) provided by the VaTeX dataset (Wang et al., 2019), in which videos were sampled at 25 fps. We initialize our multimodal model using T5 (Raffel et al., 2020), given its ability to fill in variable-length blanks. T5 is an encoder-decoder Transformer (Vaswani et al., 2017) model that is a good starting point as it provides state-of-the-art performance on text-only tasks and it was pretrained to fill arbitrary-length text spans that were previously masked. Building upon T5 allows our model to not only leverage the pre-trained large-scale language models that already have strong language abilities but also to fuse it with visual inputs. We initialize the early-fusion model with pretrained T5-base weights. For the late-fusion model, we use T5-base for the text encoder and for the decoder. We use two one-layer transformers to encode videos and fuse text and video features, and the weights of these two transformers are randomly initialized. Following T5 model implementation, the special token <extra_id_0> is used to represent the blanked phrase, and <\s> is used to separate the text and video sequences. The generated output follows T5 output format: the special token <extra_id_0> followed by the predicted text for the blanked phrase. See Appendix B.1 for more details. Most Frequent Answer. The baseline makes use of the most frequent answer in the training set (a man) as the answer to all the blanked captions during evaluation. Text-based Transformer. Previous visual question answering datasets found that a text-only model can nearly match the performance of the multimodal system (Antol et al., 2015). We analyze the degree to which language alone can contribute to our video understanding framework by conducting experiments based on text-only models. We use the off-the-shelf T5-base transformer model (Raffel et al., 2020) as our baseline model. We use both a zero-shot model (not trained on our data) and a fine-tuned model. For the latter, we use the base model v1.1 because it performed better in our experiments on the validation set. The decoding hyperparameters are the same as in the multimodal models, except the beam size is 8 for both the zero-shot one and 2 for the fine-tuned variant as we obtained the best validation results for each one using these beam sizes. Single video feature. We consider using a single I3D feature per video to determine how well the model does with a small portion of the video. Based on a study of 50 randomly sampled videos, the blanked entity in the caption appeared 95% of the time in the third second of the video (see Fig. 11 in the Appendix). For this method, we pick the I3D feature which corresponds roughly to it and apply it to the proposed multimodal methods instead of using all the video features. Note I3D takes a window of 16 frames as input, which in our case corresponds to 640 milliseconds, centered at the mentioned moment within the video. This can be seen as a small generalization of the Image Understanding task, which considers a single image (frame). We perform experiments and evaluations using the dataset described in Section 3. We use exact match accuracy and ROUGE-1 F1 score (token-level) (Lin, 2004) to evaluate the output of the generation models and to evaluate human agreement (Section 3.4). For the exact match, we count a generated text string as correct if it has at least one string-level match among the provided annotations. For the token-level F1, we compute the token overlap (true positives) between the generated text string and each annotation, normalized by the sum of the true positives and average of the false negatives/positives. We then compute the maximum across all annotations. For all evaluations, we computed the metrics based on the normalized text (i.e., without articles). We evaluate the visual understanding ability of our multimodal model by comparing its performance with the text-only baseline and the human performance. The results from the fill-in-the-blanks task are shown in Table 3. The accuracy of the text-only model and F1 score are low, indicating that the language bias is controlled in our dataset. The 2931 val test Method EM F1 EM F1 BASELINES Most Frequent Answer 15.4 45.1 16.4 45.3 T5 zero-shot 39.3 52.0 37.4 49.2 T5 fine-tuned 58.0 73.8 54.5 70.9 OUR MULTIMODAL MODELS T5 + 1f I3D 59.2 74.7 54.3 70.5 T5 + I3D 60.2 75.0 56.2 71.4 Late-fusion T5 + 1f I3D 53.7 70.3 50.3 67.6 Late-fusion T5 + I3D 53.5 69.7 51.6 67.8 UPPER BOUND (HUMANAGREEMENT ) leave one worker out 75.3 82.6 75.0 82.5 new humans* 49.0 90.2 n/a n/a Table 3: Results on the validation set. multimodal model outperforms the text-only baselines in both exact match accuracy and F1 score, meaning that our multimodal model is able to learn video features relevant to caption language during training. We also note that the early-fusion multimodal model (T5 + I3D) slightly outperforms the late-fusion multimodal model, which suggests that the model learns more effectively without extra encoders (see Fig. 4b). Both the early-fusion and the late-fusion multimodal models perform worse with a single I3D feature. This suggests that the model benefits from the whole video to correctly answer the caption. We also find a large performance gap between the multimodal model performance and the human performance. Therefore, plenty of space exists for improvements to achieve human performance, and the video fill-in-the-blanks task is worth investigating in future visual understanding research. Results per Semantic Label. To measure how well the model understands different patterns in the caption data, we compare the predictions generated for blanks corresponding to words of different semantic categories (the rest of the answers generally belong to the same category as the blanked words). Two of the authors annotated the originally blanked phrases for common non-overlapping semantic categories, including people, passive entities, and lo-Category Size (%) T5 zs T5 ft T5 + I3D Passive entity 40.4 52.9 63.6 63.6 Person 33.4 37.0 81.8 83.2 Pronoun 6.1 73.5 85.6 84.3 Location 5.5 55.1 74.5 75.4 Preposition 4.5 81.6 95.7 97.5 Action 3.9 47.8 65.5 59.9 Audio 2.5 56.4 73.0 63.6 Abstract 2.2 59.6 70.0 77.9 Other 1.5 56.9 75.0 83.7 Event 1.0 70.0 68.0 84.0 Table 4: F1 scores on the validation set for blanks with different semantic categories, in descending order based on their size. We list the categories and their distribution/size in Table 4, and we also show the performance for the best text-only zero-shot method (T5 zero-shot), text-only fine-tuned method (T5 fine-tuned), and multimodal method (T5 + I3D). The results of T5 zero-shot show some categories can be easily predicted, without fine-tuning on the dataset, namely Preposition , Pronoun , and Event . However, fine-tuning T5 on our dataset yields improvements for nearly all categories. The multimodal (T5 + I3D) model improves the categories of Person and Abstract nouns but performs worse for others, namely Audio and Action . This finding follows from the fact that understanding higher-order audio and visual concepts requires complex reasoning, for which the video-aware model may need more training. In general, Action and Passive entity will likely require extra attention in future work, considering the comparatively low performance for these categories. gain insights on how to improve our models for future work, we measure where our best model (T5 + I3D) fails and humans perform well. We find three main types of wrong predictions. The most common error is predicting man instead of women, followed by predicting person instead of child or baby. The majority of the remaining errors are predictions close to the ground truth answers such as dance instead of exercise, pillow instead of sheets, rug instead of sand, floor instead of court, knife instead of spatula or basketball game instead of wrestling. Based on these types of errors, in future work, the model would benefit from pre-training on unbiased data (both gender and age) and also from pre-training on a large-scale multimodal (language and video) dataset, to learn about more diverse situations and objects. This paper introduced the fill-in-the-blanks evaluation framework for video understanding. The framework addresses drawbacks of alternative video understanding tasks, such as multiple-choice visual question answering or video captioning. Our paper makes three important contributions. First, we introduced FIBER , which is a large dataset consisting of 28,000 videos and tests based on filling blanks, building upon an existing video captioning dataset with a new set of manual annotations, and using a modified annotation framework to encourage diverse responses among annotators. This process can be easily replicated to create new fill-in-the-blanks data for other datasets and tasks. Second, we conducted extensive analyses on the dataset to evaluate the quality of the annotations and to understand the patterns and limitations of the data. Finally, we introduced a multimodal model that fuses language and visual information and found that the video-aware models significantly outperform the text-only models. Notably, we found a consistent gap between model performance and human performance, which suggests room for improvement in future models addressing video understanding through the lens of the fill-in-the-blanks task. Even though we compensated the annotators based on the quality of the answers they produced (and stated so in the instructions), they were rewarded based on the number of answers they input since we looked for diversity. These incentives may have encouraged the annotators to make many judgments quickly and therefore make biased decisions. Due to these biases, we cannot guarantee that annota-tors' guesses always match reality. Based on spot-checking, it seems that annotators made reasonable judgments, but others may disagree. We have also observed our data is skewed toward more male noun phrases (cf. Appendix A.5), which could be due to a bias both in VaTeX and in the annotators we hired. Our evaluation weights all errors equally, even though some errors may have a bigger impact than others. For example, someone in a video may be misgendered by being referred to as a man when the correct reference should be woman. We thank Laura Biester for helping with data quality assurance. We thank the following people for reviewing drafts of this document: Artem Abza-liev, Christine Feak, Victoria Florence, Zhijing Jin, and Max Krogius. We also want to thank the LIT Research Group @ UMich members for feedback on some of the ideas discussed here. This material is based in part upon work supported by the Automotive Research Center (ARC). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of ARC or any other related entity.
objective abstain other other abstain abstain abstain abstain abstain abstain objective method method abstain other abstain result abstain result objective objective objective objective other other other other other other other other other other other other other other other other method other other method method method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain abstain abstain abstain method method method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain abstain method method abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method method abstain abstain abstain method method abstain method method method method abstain abstain abstain abstain abstain abstain abstain abstain method method method abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result result abstain abstain result abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective objective abstain method result result method abstain abstain abstain result abstain abstain other other other other other