Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling
<|reference_start|>Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling: Today's most accurate language models are trained on orders of magnitude more language data than human language learners receive - but with no supervision from other sensory modalities that play a crucial role in human learning. Can we make LMs' representations and predictions more accurate (and more human-like) with more ecologically plausible supervision? This paper describes LexiContrastive Grounding (LCG), a grounded language learning procedure that leverages visual supervision to improve textual representations. LexiContrastive Grounding combines a next token prediction strategy with a contrastive visual grounding objective, focusing on early-layer representations that encode lexical information. Across multiple word-learning and sentence-understanding benchmarks, LexiContrastive Grounding not only outperforms standard language-only models in learning efficiency, but also improves upon vision-and-language learning procedures including CLIP, GIT, Flamingo, and Vokenization. Moreover, LexiContrastive Grounding improves perplexity by around 5% on multiple language modeling tasks. This work underscores the potential of incorporating visual grounding into language models, aligning more closely with the multimodal nature of human language acquisition.<|reference_end|>
@article{zhuang2024lexicon-level,
    title={Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling},
    author={Chengxu Zhuang, Evelina Fedorenko, Jacob Andreas},
    journal={arXiv preprint arXiv:2403.14551},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.14551},
    primaryClass={cs.CL cs.AI cs.LG}
}