Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction
<|reference_start|>Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction: Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.<|reference_end|>
@article{kumar2024comparing,
    title={Comparing Abstraction in Humans and Large Language Models Using
  Multimodal Serial Reproduction},
    author={Sreejan Kumar, Raja Marjieh, Byron Zhang, Declan Campbell, Michael Y.
  Hu, Umang Bhatt, Brenden Lake, Thomas L. Griffiths},
    journal={arXiv preprint arXiv:2402.03618},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03618},
    primaryClass={cs.AI cs.CL q-bio.NC}
}