title: Do great minds think alike? Investigating Human-AI Complementarity in
    Question Answering with CAIMIRA
summary: Recent advancements of large language models (LLMs) have led to claims of AI
  surpassing humans in natural language processing (NLP) tasks such as textual
  understanding and reasoning. This work investigates these assertions by
  introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
  that enables quantitative assessment and comparison of problem-solving
  abilities of question-answering (QA) agents: humans and AI systems. Through
  analysis of over 300,000 responses from ~70 AI systems and 155 humans across
  thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
  knowledge domains and reasoning skills. Humans outperform AI systems in
  knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
  LLMs like GPT-4 and LLaMA show superior performance on targeted information
  retrieval and fact-based reasoning, particularly when information gaps are
  well-defined and addressable through pattern matching or data retrieval. These
  findings highlight the need for future QA tasks to focus on questions that
  challenge not only higher-order reasoning and scientific thinking, but also
  demand nuanced linguistic interpretation and cross-contextual knowledge
  application, helping advance AI developments that better emulate or complement
  human cognitive abilities in real-world problem-solving.
paper_authors._id: 67074de616abdf6976c85bdf
paper_authors.name: Maharshi Gor
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:43:21.613Z
paper_authors._id: 639410ddcd5e6b3cdc4ca80e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/639410ddcd5e6b3cdc4ca80e/Gllj99qgxh6Gm__p7sizf.jpeg
paper_authors.fullname: Maharshi Gor
paper_authors.type: user
paper_authors.user: mgor
paper_authors._id: 67074de616abdf6976c85be0
paper_authors.name: Hal Daumé III
paper_authors._id: 67074de616abdf6976c85be1
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:00:42.432Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 67074de616abdf6976c85be2
paper_authors.name: Jordan Boyd-Graber
paper_title: Do great minds think alike? Investigating Human-AI Complementarity in
    Question Answering with CAIMIRA
paper_summary: Recent advancements of large language models (LLMs) have led to claims of AI
  surpassing humans in natural language processing (NLP) tasks such as textual
  understanding and reasoning. This work investigates these assertions by
  introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
  that enables quantitative assessment and comparison of problem-solving
  abilities of question-answering (QA) agents: humans and AI systems. Through
  analysis of over 300,000 responses from ~70 AI systems and 155 humans across
  thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
  knowledge domains and reasoning skills. Humans outperform AI systems in
  knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
  LLMs like GPT-4 and LLaMA show superior performance on targeted information
  retrieval and fact-based reasoning, particularly when information gaps are
  well-defined and addressable through pattern matching or data retrieval. These
  findings highlight the need for future QA tasks to focus on questions that
  challenge not only higher-order reasoning and scientific thinking, but also
  demand nuanced linguistic interpretation and cross-contextual knowledge
  application, helping advance AI developments that better emulate or complement
  human cognitive abilities in real-world problem-solving.
paper_ai_keywords: item response theory (IRT) knowledge-grounded abductive and conceptual reasoning targeted information retrieval fact-based reasoning pattern matching data retrieval higher-order reasoning scientific thinking nuanced linguistic interpretation cross-contextual knowledge application