  Linguistic representations derived from text alone have been criticized for
their lack of grounding, i.e., connecting words to their meanings in the
physical world. Vision-and-Language (VL) models, trained jointly on text and
image or video data, have been offered as a response to such criticisms.
However, while VL pretraining has shown success on multimodal tasks such as
visual question answering, it is not yet known how the internal linguistic
representations themselves compare to their text-only counterparts. This paper
compares the semantic representations learned via VL vs. text-only pretraining
for two recent VL models using a suite of analyses (clustering, probing, and
performance on a commonsense question answering task) in a language-only
setting. We find that the multimodal models fail to significantly outperform
the text-only variants, suggesting that future work is required if multimodal
pretraining is to be pursued as a means of improving NLP in general.

Computation and Language;Artificial Intelligence;Computer Vision and Pattern Recognition