  This paper focuses on analyzing and improving the commonsense ability of
recent popular vision-language (VL) models. Despite the great success, we
observe that existing VL-models still lack commonsense knowledge/reasoning
ability (e.g., "Lemons are sour"), which is a vital component towards
artificial general intelligence. Through our analysis, we find one important
reason is that existing large-scale VL datasets do not contain much commonsense
knowledge, which motivates us to improve the commonsense of VL-models from the
data perspective. Rather than collecting a new VL training dataset, we propose
a more scalable strategy, i.e., "Data Augmentation with kNowledge graph
linearization for CommonsensE capability" (DANCE). It can be viewed as one type
of data augmentation technique, which can inject commonsense knowledge into
existing VL datasets on the fly during training. More specifically, we leverage
the commonsense knowledge graph (e.g., ConceptNet) and create variants of text
description in VL datasets via bidirectional sub-graph sequentialization. For
better commonsense evaluation, we further propose the first retrieval-based
commonsense diagnostic benchmark. By conducting extensive experiments on some
representative VL-models, we demonstrate that our DANCE technique is able to
significantly improve the commonsense ability while maintaining the performance
on vanilla retrieval tasks. The code and data are available at
https://github.com/pleaseconnectwifi/DANCE

Computer Vision and Pattern Recognition;Artificial Intelligence;Machine Learning