title: Comparing Machines and Children: Using Developmental Psychology
    Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
summary: Developmental psychologists have spent decades devising experiments to test
  the intelligence and knowledge of infants and children, tracing the origin of
  crucial concepts and capacities. Moreover, experimental techniques in
  developmental psychology have been carefully designed to discriminate the
  cognitive capacities that underlie particular behaviors. We propose that using
  classical experiments from child development is a particularly effective way to
  probe the computational abilities of AI models, in general, and LLMs in
  particular. First, the methodological techniques of developmental psychology,
  such as the use of novel stimuli to control for past experience or control
  conditions to determine whether children are using simple associations, can be
  equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
  in this way can tell us whether the information that is encoded in text is
  sufficient to enable particular responses, or whether those responses depend on
  other kinds of information, such as information from exploration of the
  physical world. In this work we adapt classical developmental experiments to
  evaluate the capabilities of LaMDA, a large language model from Google. We
  propose a novel LLM Response Score (LRS) metric which can be used to evaluate
  other language models, such as GPT. We find that LaMDA generates appropriate
  responses that are similar to those of children in experiments involving social
  understanding, perhaps providing evidence that knowledge of these domains is
  discovered through language. On the other hand, LaMDA's responses in early
  object and action understanding, theory of mind, and especially causal
  reasoning tasks are very different from those of young children, perhaps
  showing that these domains require more real-world, self-initiated exploration
  and cannot simply be learned from patterns in language input.
paper_authors._id: 646ac83c31fbfc5df81f7cde
paper_authors.name: Eliza Kosoy
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-22T18:01:31.951Z
paper_authors._id: 646b8b60c7f672003c84a876
paper_authors.avatarUrl: /avatars/9436252f4dee43f462f1363e3b899915.svg
paper_authors.fullname: Eliza Kosoy
paper_authors.type: user
paper_authors.user: Mathpun314
paper_authors._id: 646ac83c31fbfc5df81f7cdf
paper_authors.name: Emily Rose Reagan
paper_authors._id: 646ac83c31fbfc5df81f7ce0
paper_authors.name: Leslie Lai
paper_authors._id: 646ac83c31fbfc5df81f7ce1
paper_authors.name: Alison Gopnik
paper_authors._id: 646ac83c31fbfc5df81f7ce2
paper_authors.name: Danielle Krettek Cobb
paper_title: Comparing Machines and Children: Using Developmental Psychology
    Experiments to Assess the Strengths and Weaknesses of LaMDA Responses
paper_summary: Developmental psychologists have spent decades devising experiments to test
  the intelligence and knowledge of infants and children, tracing the origin of
  crucial concepts and capacities. Moreover, experimental techniques in
  developmental psychology have been carefully designed to discriminate the
  cognitive capacities that underlie particular behaviors. We propose that using
  classical experiments from child development is a particularly effective way to
  probe the computational abilities of AI models, in general, and LLMs in
  particular. First, the methodological techniques of developmental psychology,
  such as the use of novel stimuli to control for past experience or control
  conditions to determine whether children are using simple associations, can be
  equally helpful for assessing the capacities of LLMs. In parallel, testing LLMs
  in this way can tell us whether the information that is encoded in text is
  sufficient to enable particular responses, or whether those responses depend on
  other kinds of information, such as information from exploration of the
  physical world. In this work we adapt classical developmental experiments to
  evaluate the capabilities of LaMDA, a large language model from Google. We
  propose a novel LLM Response Score (LRS) metric which can be used to evaluate
  other language models, such as GPT. We find that LaMDA generates appropriate
  responses that are similar to those of children in experiments involving social
  understanding, perhaps providing evidence that knowledge of these domains is
  discovered through language. On the other hand, LaMDA's responses in early
  object and action understanding, theory of mind, and especially causal
  reasoning tasks are very different from those of young children, perhaps
  showing that these domains require more real-world, self-initiated exploration
  and cannot simply be learned from patterns in language input.
paper_ai_keywords: large language model (LLM) Discriminative abilities Computational abilities Cognitive capacities LLM Response Score (LRS) LaMDA Social understanding Object and action understanding Theory of mind Causal reasoning