Do great minds think alike? Investigating Human-AI Complementarity in
  Question Answering with CAIMIRA
Recent advancements of large language models (LLMs) have led to claims of AI
surpassing humans in natural language processing (NLP) tasks such as textual
understanding and reasoning. This work investigates these assertions by
introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
that enables quantitative assessment and comparison of problem-solving
abilities of question-answering (QA) agents: humans and AI systems. Through
analysis of over 300,000 responses from ~70 AI systems and 155 humans across
thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
knowledge domains and reasoning skills. Humans outperform AI systems in
knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
LLMs like GPT-4 and LLaMA show superior performance on targeted information
retrieval and fact-based reasoning, particularly when information gaps are
well-defined and addressable through pattern matching or data retrieval. These
findings highlight the need for future QA tasks to focus on questions that
challenge not only higher-order reasoning and scientific thinking, but also
demand nuanced linguistic interpretation and cross-contextual knowledge
application, helping advance AI developments that better emulate or complement
human cognitive abilities in real-world problem-solving.
67074de616abdf6976c85bdf
Maharshi Gor
claimed_verified
2024-10-10T08:43:21.613Z
639410ddcd5e6b3cdc4ca80e
https://cdn-avatars.huggingface.co/v1/production/uploads/639410ddcd5e6b3cdc4ca80e/Gllj99qgxh6Gm__p7sizf.jpeg
Maharshi Gor
user
mgor
67074de616abdf6976c85be0
Hal Daumé III
67074de616abdf6976c85be1
Tianyi Zhou
claimed_verified
2024-10-10T08:00:42.432Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
67074de616abdf6976c85be2
Jordan Boyd-Graber
Do great minds think alike? Investigating Human-AI Complementarity in
  Question Answering with CAIMIRA
Recent advancements of large language models (LLMs) have led to claims of AI
surpassing humans in natural language processing (NLP) tasks such as textual
understanding and reasoning. This work investigates these assertions by
introducing CAIMIRA, a novel framework rooted in item response theory (IRT)
that enables quantitative assessment and comparison of problem-solving
abilities of question-answering (QA) agents: humans and AI systems. Through
analysis of over 300,000 responses from ~70 AI systems and 155 humans across
thousands of quiz questions, CAIMIRA uncovers distinct proficiency patterns in
knowledge domains and reasoning skills. Humans outperform AI systems in
knowledge-grounded abductive and conceptual reasoning, while state-of-the-art
LLMs like GPT-4 and LLaMA show superior performance on targeted information
retrieval and fact-based reasoning, particularly when information gaps are
well-defined and addressable through pattern matching or data retrieval. These
findings highlight the need for future QA tasks to focus on questions that
challenge not only higher-order reasoning and scientific thinking, but also
demand nuanced linguistic interpretation and cross-contextual knowledge
application, helping advance AI developments that better emulate or complement
human cognitive abilities in real-world problem-solving.
CAIMIRA, a framework based on item response theory, quantitatively assesses and compares the problem-solving abilities of humans and AI systems across various knowledge domains and reasoning skills, revealing strengths and weaknesses in reasoning types.
item response theory question-answering agents knowledge-grounded abductive reasoning conceptual reasoning targeted information retrieval fact-based reasoning pattern matching data retrieval higher-order reasoning scientific thinking nuanced linguistic interpretation cross-contextual knowledge application