Comparing Machines and Children: Using Developmental Psychology
  Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
Developmental psychologists have spent decades devising experiments to test
the intelligence and knowledge of infants and children, tracing the origin of
crucial concepts and capacities. Moreover, experimental techniques in
developmental psychology have been carefully designed to discriminate the
cognitive capacities that underlie particular behaviors. We propose that using
classical experiments from child development is a particularly effective way to
probe the computational abilities of AI models, in general, and LLMs in
particular. First, the methodological techniques of developmental psychology,
such as the use of novel stimuli to control for past experience or control
conditions to determine whether children are using simple associations, can be
equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
in this way can tell us whether the information that is encoded in text is
sufficient to enable particular responses, or whether those responses depend on
other kinds of information, such as information from exploration of the
physical world. In this work we adapt classical developmental experiments to
evaluate the capabilities of LaMDA, a large language model from Google. We
propose a novel LLM Response Score (LRS) metric which can be used to evaluate
other language models, such as GPT. We find that LaMDA generates appropriate
responses that are similar to those of children in experiments involving social
understanding, perhaps providing evidence that knowledge of these domains is
discovered through language. On the other hand, LaMDA's responses in early
object and action understanding, theory of mind, and especially causal
reasoning tasks are very different from those of young children, perhaps
showing that these domains require more real-world, self-initiated exploration
and cannot simply be learned from patterns in language input.
646ac83c31fbfc5df81f7cde
Eliza Kosoy
claimed_verified
2023-05-22T18:01:31.951Z
646b8b60c7f672003c84a876
/avatars/9436252f4dee43f462f1363e3b899915.svg
Eliza Kosoy
user
Mathpun314
646ac83c31fbfc5df81f7cdf
Emily Rose Reagan
646ac83c31fbfc5df81f7ce0
Leslie Lai
646ac83c31fbfc5df81f7ce1
Alison Gopnik
646ac83c31fbfc5df81f7ce2
Danielle Krettek Cobb
Comparing Machines and Children: Using Developmental Psychology
  Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
Developmental psychologists have spent decades devising experiments to test
the intelligence and knowledge of infants and children, tracing the origin of
crucial concepts and capacities. Moreover, experimental techniques in
developmental psychology have been carefully designed to discriminate the
cognitive capacities that underlie particular behaviors. We propose that using
classical experiments from child development is a particularly effective way to
probe the computational abilities of AI models, in general, and LLMs in
particular. First, the methodological techniques of developmental psychology,
such as the use of novel stimuli to control for past experience or control
conditions to determine whether children are using simple associations, can be
equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
in this way can tell us whether the information that is encoded in text is
sufficient to enable particular responses, or whether those responses depend on
other kinds of information, such as information from exploration of the
physical world. In this work we adapt classical developmental experiments to
evaluate the capabilities of LaMDA, a large language model from Google. We
propose a novel LLM Response Score (LRS) metric which can be used to evaluate
other language models, such as GPT. We find that LaMDA generates appropriate
responses that are similar to those of children in experiments involving social
understanding, perhaps providing evidence that knowledge of these domains is
discovered through language. On the other hand, LaMDA's responses in early
object and action understanding, theory of mind, and especially causal
reasoning tasks are very different from those of young children, perhaps
showing that these domains require more real-world, self-initiated exploration
and cannot simply be learned from patterns in language input.
Classical developmental psychology experiments adapted to evaluate AI models reveal that while language models can mimic social understanding, they struggle with early object and action understanding, theory of mind, and causal reasoning, suggesting the need for real-world experience.
developmental psychology experimental techniques cognitive capacities computational abilities AI models LLMs LaMDA LLM Response Score (LRS) social understanding theory of mind causal reasoning