  Large-scale pre-training has recently revolutionized vision-and-language (VL)
research. Models such as LXMERT and UNITER have significantly lifted the state
of the art over a wide range of VL tasks. However, the large number of
parameters in such models hinders their application in practice. In parallel,
work on the lottery ticket hypothesis (LTH) has shown that deep neural networks
contain small matching subnetworks that can achieve on par or even better
performance than the dense networks when trained in isolation. In this work, we
perform the first empirical study to assess whether such trainable subnetworks
also exist in pre-trained VL models. We use UNITER as the main testbed (also
test on LXMERT and ViLT), and consolidate 7 representative VL tasks for
experiments, including visual question answering, visual commonsense reasoning,
visual entailment, referring expression comprehension, image-text retrieval,
GQA, and NLVR$^2$. Through comprehensive analysis, we summarize our main
findings as follows. ($i$) It is difficult to find subnetworks that strictly
match the performance of the full model. However, we can find "relaxed" winning
tickets at 50%-70% sparsity that maintain 99% of the full accuracy. ($ii$)
Subnetworks found by task-specific pruning transfer reasonably well to the
other tasks, while those found on the pre-training tasks at 60%/70% sparsity
transfer universally, matching 98%/96% of the full accuracy on average over all
the tasks. ($iii$) Besides UNITER, other models such as LXMERT and ViLT can
also play lottery tickets. However, the highest sparsity we can achieve for
ViLT is far lower than LXMERT and UNITER (30% vs. 70%). ($iv$) LTH also remains
relevant when using other training methods (e.g., adversarial training).

Computer Vision and Pattern Recognition;Computation and Language;Machine Learning