While large-scale  language models have enjoyed great success recently, much remains to be understood about what is encoded in their representations. In this work, we propose a method for characterizing how language representations of concrete nouns  relate to the physical appearance of the objects they refer to. Our approach uses a probing model that examines how useful language representations are in discerning between different visual representations. We show evidence of a surprising common ground with the visual domain, finding representations of many language models to be useful in retrieving semantically aligned image patches . In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval. Finally, we show that the examined models substantially under-perform humans in retrieval  . Altogether, our findings shed new empirical insights on language grounding , suggesting that some physical properties are being captured by trained language models, and highlighting large room for future progress .
While large-scale contextual language models have enjoyed great success recently, much remains to be understood about what is encoded in their representations. In this work, we characterize how contextual representations of concrete nouns extracted by trained language models relate to the physical properties of the objects they refer to. Our approach uses a probing model that examines how effective these language representations are in discerning between different visual representations. We show that many recent language models yield representations that are useful in retrieving semantically aligned image patches , and explore the role of context in this process. Much weaker results are found in control experiments, attesting the selectivity of the probe. All examined models greatly under-perform humans in retrieval , highlighting substantial room for future progress . Altogether, our findings shed new empirical insights on language grounding and its materialization in contextual language models .
A
contextual
meaning-changed
meaning-changed coherence meaning-changed
R
propose a method for characterizing how language
characterize how contextual
clarity
clarity clarity clarity
A
extracted by trained language models
meaning-changed
meaning-changed meaning-changed meaning-changed
R
appearance
properties
clarity
clarity clarity clarity
R
useful
effective these
style
style style style
R
evidence of a surprising common ground with the visual domain, finding representations of many language models to be
that many recent language models yield representations that are
clarity
clarity clarity clarity
R
. In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval. Finally, we show that the examined models substantially
, and explore the role of context in this process. Much weaker results are found in control experiments, attesting the selectivity of the probe. All examined models greatly
clarity
meaning-changed clarity clarity
A
, highlighting substantial room for future progress
meaning-changed
meaning-changed meaning-changed coherence
R
, suggesting that some physical properties are being captured by trained language models, and highlighting large room for future progress
and its materialization in contextual language models
clarity
meaning-changed clarity clarity