  We propose VALSE (Vision And Language Structured Evaluation), a novel
benchmark designed for testing general-purpose pretrained vision and language
(V&L) models for their visio-linguistic grounding capabilities on specific
linguistic phenomena. VALSE offers a suite of six tests covering various
linguistic constructs. Solving these requires models to ground linguistic
phenomena in the visual modality, allowing more fine-grained evaluations than
hitherto possible. We build VALSE using methods that support the construction
of valid foils, and report results from evaluating five widely-used V&L models.
Our experiments suggest that current models have considerable difficulty
addressing most phenomena. Hence, we expect VALSE to serve as an important
benchmark to measure future progress of pretrained V&L models from a linguistic
perspective, complementing the canonical task-centred V&L evaluations.

Computation and Language;Computer Vision and Pattern Recognition