61A-Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff (Now What?)
<|reference_start|>61A-Bot Report: AI Assistants in CS1 Save Students Homework Time and Reduce Demands on Staff (Now What?): Chatbot interfaces for LLMs enable students to get immediate, interactive help on homework assignments, but even a thoughtfully-designed bot may not serve all pedagogical goals. In this paper, we report on the development and deployment of a GPT-4-based interactive homework assistant ("61A Bot") for students in a large CS1 course; over 2000 students made over 100,000 requests of our bot across two semesters. Our assistant offers one-shot, contextual feedback, primarily through a low-friction "get feedback" prompt within the command-line "autograder" our students already run to test their code. Our Bot wraps student code in a custom prompt that supports our pedagogical goals and avoids providing solutions directly. We discuss our deployment and then analyze the impacts of our Bot on students, primarily through student-reported feedback and tracking of student homework progress. We find reductions in homework-related question rates in our course forum, as well as substantial reductions in homework completion time when our Bot is available. For students in the 50th-80th percentile, these reductions typically exceed 30 minutes per assignment, over 4 standard deviations faster than the mean in prior semesters. Finally, we conclude with a discussion of these observations, the potential impacts on student learning, as well as other potential costs and benefits of AI assistance in CS1.<|reference_end|>
@article{zamfirescu-pereira202461a-bot,
    title={61A-Bot Report: AI Assistants in CS1 Save Students Homework Time and
  Reduce Demands on Staff. (Now What?)},
    author={J.D. Zamfirescu-Pereira, Laryn Qi, Bj"orn Hartmann, John DeNero,
  Narges Norouzi},
    journal={arXiv preprint arXiv:2406.05600},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.05600},
    primaryClass={cs.CY}
}