Vision, as a central component of human perception, plays a fundamental role in shaping natural language . To better understand how text models are connected to our visual perceptions , we propose a method for examining the similarities between neural representations extracted from words in text and objects in images . Our approach uses a lightweight probing model that learns to map language representations of concrete words to the visual domain . We find that representations from models trained on purely textual data, such as BERT, can be nontrivially mapped to those of a vision model. Such mappings generalize to object categories that were never seen by the probe during training, unlike mappings learned from permuted or random representations . Moreover, we find that the context surrounding objects in sentences greatly impacts performance. Finally, we show that humans significantly outperform all examined models , suggesting considerable room for improvement in representation learning and grounding .
While large-scale language models have enjoyed great success recently, much remains to be understood about what is encoded in their representations. In this work , we propose a method for characterizing how language representations of concrete nouns relate to the physical appearance of the objects they refer to . Our approach uses a probing model that examines how useful language representations are in discerning between different visual representations. We show evidence of a surprising common ground with the visual domain , finding representations of many language models to be useful in retrieving semantically aligned image patches. In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval . Finally, we show that the examined models substantially under-perform humans in retrieval. Altogether, our findings shed new empirical insights on language grounding, suggesting that some physical properties are being captured by trained language models, and highlighting large room for future progress .
R
Vision, as a central component of human perception, plays a fundamental role in shaping natural language . To better understand how text models are connected to our visual perceptions
While large-scale language models have enjoyed great success recently, much remains to be understood about what is encoded in their representations. In this work
clarity
clarity clarity meaning-changed
R
examining the similarities between neural representations extracted from words in text and objects in images
characterizing how language representations of concrete nouns relate to the physical appearance of the objects they refer to
meaning-changed
meaning-changed meaning-changed clarity
D
lightweight
coherence
fluency coherence coherence
R
learns to map language representations of concrete words to
examines how useful language representations are in discerning between different visual representations. We show evidence of a surprising common ground with
clarity
clarity clarity clarity
R
. We find that representations from models trained on purely textual data, such as BERT, can be nontrivially mapped to those of a vision model. Such mappings generalize to object categories that were never seen by the probe during training, unlike mappings learned from permuted or random representations
, finding representations of many language models to be useful in retrieving semantically aligned image patches. In control experiments where language and visual representations are intentionally mismatched, we observe much weaker results. Furthermore, we examine the impact of textual context in our experiments, finding, for instance, that nouns accompanied by adjectives lead to more accurate retrieval
meaning-changed
meaning-changed meaning-changed meaning-changed
D
Moreover, we find that the context surrounding objects in sentences greatly impacts performance.
clarity
clarity clarity others
R
humans significantly outperform all examined models , suggesting considerable room for improvement in representation learning and grounding
the examined models substantially under-perform humans in retrieval. Altogether, our findings shed new empirical insights on language grounding, suggesting that some physical properties are being captured by trained language models, and highlighting large room for future progress
clarity
clarity meaning-changed clarity