While large-scale contextual language models have enjoyed great success recently, much remains to be understood about what is encoded in their representations. In this work, we characterize how contextual representations of concrete nouns extracted by trained language models relate to the physical properties of the objects they refer to. Our approach uses a probing model that examines how effective these language representations are in discerning between different visual representations. We show that many recent language models yield representations that are useful in retrieving semantically aligned image patches , and explore the role of context in this process. Much weaker results are found in control experiments, attesting the selectivity of the probe. All examined models greatly under-perform humans in retrieval, highlighting substantial room for future progress. Altogether, our findings shed new empirical insights on language grounding and its materialization in contextual language models.
The success of large-scale contextual language models has attracted great interest in probing what is encoded in their representations. In this work, we consider a new question: to what extent contextual representations of concrete nouns are aligned with corresponding visual representations? We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role in this process. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans . We hope our analyses inspire future research in understanding and improving the visual capabilities of language models.
R
While
The success of
R
have enjoyed great success recently, much remains to be understood about
has attracted great interest in probing
R
characterize how
consider a new question: to what extent
R
extracted by trained language models relate to the physical properties of the objects they refer to. Our approach uses
are aligned with corresponding visual representations? We design
R
examines how effective these language representations are in discerning between different
evaluates how effective are text-only representations in distinguishing between matching and non-matching
R
We show that many recent language models yield representations that are useful in retrieving semantically aligned image patches , and explore the role of context
Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Moreover, they are effective in retrieving specific instances of image patches; textual context plays an important role
R
Much weaker results are found in control experiments, attesting the selectivity of the probe. All examined models
Visually grounded language models slightly outperform text-only language models in instance retrieval, but
R
in retrieval, highlighting substantial room for future progress. Altogether, our findings shed new empirical insights on language grounding and its materialization in contextual
. We hope our analyses inspire future research in understanding and improving the visual capabilities of