Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Generative AI, particularly Language Models (LMs), has the potential to
transform real-world domains with societal impact, particularly where access to
experts is limited. For example, in education, training novice educators with
expert guidance is important for effectiveness but expensive, creating
significant barriers to improving education quality at scale. This challenge
disproportionately harms students from under-served communities, who stand to
gain the most from high-quality education. We introduce Tutor CoPilot, a novel
Human-AI approach that leverages a model of expert thinking to provide
expert-like guidance to tutors as they tutor. This study is the first
randomized controlled trial of a Human-AI system in live tutoring, involving
900 tutors and 1,800 K-12 students from historically under-served communities.
Following a preregistered analysis plan, we find that students working with
tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more
likely to master topics (p<0.01). Notably, students of lower-rated tutors
experienced the greatest benefit, improving mastery by 9 p.p. We find that
Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages
using classifiers to identify pedagogical strategies, and find that tutors with
access to Tutor CoPilot are more likely to use high-quality strategies to
foster student understanding (e.g., asking guiding questions) and less likely
to give away the answer to the student. Tutor interviews highlight how Tutor
CoPilot's guidance helps tutors to respond to student needs, though they flag
issues in Tutor CoPilot, such as generating suggestions that are not
grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates
how Human-AI systems can scale expertise in real-world domains, bridge gaps in
skills and create a future where high-quality education is accessible to all
students.
6703648177f7e874a2a560be
Rose E. Wang
claimed_verified
2024-10-07T07:26:30.010Z
651e391852cd39f995249099
/avatars/59d295863c13f90cb345d5cac303c266.svg
Rose E Wang
user
rose-e-wang
6703648177f7e874a2a560bf
Ana T. Ribeiro
admin_assigned
2024-10-07T08:56:43.863Z
639234b5c1899a08e56a7d2a
/avatars/34829332a4b10cafab77e396e3c57752.svg
Ana Ribeiro
user
Analu
6703648177f7e874a2a560c0
Carly D. Robinson
admin_assigned
2024-10-07T08:56:52.843Z
660c6f8b0c47aba6600c247e
/avatars/7f79e19f6c8186cdb40785140cd42578.svg
Carly Robinson
user
carlycodes
6703648177f7e874a2a560c1
Susanna Loeb
6703648177f7e874a2a560c2
Dora Demszky
admin_assigned
2024-10-07T08:57:02.827Z
606e3a4c7f3150d181025dd1
/avatars/224ee859c1df3f6c4ab59910f2a4732b.svg
Dora Demszky
user
ddemszky
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Generative AI, particularly Language Models (LMs), has the potential to
transform real-world domains with societal impact, particularly where access to
experts is limited. For example, in education, training novice educators with
expert guidance is important for effectiveness but expensive, creating
significant barriers to improving education quality at scale. This challenge
disproportionately harms students from under-served communities, who stand to
gain the most from high-quality education. We introduce Tutor CoPilot, a novel
Human-AI approach that leverages a model of expert thinking to provide
expert-like guidance to tutors as they tutor. This study is the first
randomized controlled trial of a Human-AI system in live tutoring, involving
900 tutors and 1,800 K-12 students from historically under-served communities.
Following a preregistered analysis plan, we find that students working with
tutors that have access to Tutor CoPilot are 4 percentage points (p.p.) more
likely to master topics (p<0.01). Notably, students of lower-rated tutors
experienced the greatest benefit, improving mastery by 9 p.p. We find that
Tutor CoPilot costs only $20 per-tutor annually. We analyze 550,000+ messages
using classifiers to identify pedagogical strategies, and find that tutors with
access to Tutor CoPilot are more likely to use high-quality strategies to
foster student understanding (e.g., asking guiding questions) and less likely
to give away the answer to the student. Tutor interviews highlight how Tutor
CoPilot's guidance helps tutors to respond to student needs, though they flag
issues in Tutor CoPilot, such as generating suggestions that are not
grade-level appropriate. Altogether, our study of Tutor CoPilot demonstrates
how Human-AI systems can scale expertise in real-world domains, bridge gaps in
skills and create a future where high-quality education is accessible to all
students.
Tutor CoPilot, a Human-AI system, enhances tutoring effectiveness by providing expert-like guidance, resulting in better student mastery and improved pedagogical strategies.