title: Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4,
    and Human Tutors
summary: Generative AI and large language models hold great promise in enhancing
  computing education by powering next-generation educational technologies for
  introductory programming. Recent works have studied these models for different
  scenarios relevant to programming education; however, these works are limited
  for several reasons, as they typically consider already outdated models or only
  specific scenario(s). Consequently, there is a lack of a systematic study that
  benchmarks state-of-the-art models for a comprehensive set of programming
  education scenarios. In our work, we systematically evaluate two models,
  ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human
  tutors for a variety of scenarios. We evaluate using five introductory Python
  programming problems and real-world buggy programs from an online platform, and
  assess performance using expert-based annotations. Our results show that GPT-4
  drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human
  tutors' performance for several scenarios. These results also highlight
  settings where GPT-4 still struggles, providing exciting future directions on
  developing techniques to improve the performance of these models.
paper_authors._id: 649e2f2c27145c446312a322
paper_authors.name: Tung Phung
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T11:47:13.186Z
paper_authors._id: 64a27d5bcd362b2c08cdf5d1
paper_authors.avatarUrl: /avatars/f3eda4e7811c15c326a38dfa18b534c2.svg
paper_authors.fullname: Minh Tung Phung
paper_authors.type: user
paper_authors.user: tungmphung
paper_authors._id: 649e2f2c27145c446312a323
paper_authors.name: Victor-Alexandru Pădurean
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T07:27:33.716Z
paper_authors._id: 649fe1259b6e48cd6b4d8615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649fe1259b6e48cd6b4d8615/Fz89v0I7EyxKKsRe4rwRt.jpeg
paper_authors.fullname: Victor Padurean
paper_authors.type: user
paper_authors.user: vicondrus
paper_authors._id: 649e2f2c27145c446312a324
paper_authors.name: José Cambronero
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T07:27:45.201Z
paper_authors._id: 649f4bfbb469b6b0b402b082
paper_authors.avatarUrl: /avatars/25eefad619629fefb85334e14cf9bd47.svg
paper_authors.fullname: Jose Cambronero
paper_authors.type: user
paper_authors.user: josepablocam
paper_authors._id: 649e2f2c27145c446312a325
paper_authors.name: Sumit Gulwani
paper_authors._id: 649e2f2c27145c446312a326
paper_authors.name: Tobias Kohn
paper_authors._id: 649e2f2c27145c446312a327
paper_authors.name: Rupak Majumdar
paper_authors._id: 649e2f2c27145c446312a328
paper_authors.name: Adish Singla
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T07:27:39.952Z
paper_authors._id: 649f535d9567d0d9fb75522f
paper_authors.avatarUrl: /avatars/161382abfa73ee354d05eee9b468e058.svg
paper_authors.fullname: Adish Singla
paper_authors.type: user
paper_authors.user: adishsingla
paper_authors._id: 649e2f2c27145c446312a329
paper_authors.name: Gustavo Soares
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2023-06-30T01:26:04.541Z
paper_authors._id: 63a605226fb8987e8bb97d3b
paper_authors.avatarUrl: /avatars/c6c8bc37f12d0593462099f2ea5cfe73.svg
paper_authors.fullname: Gustavo Soares
paper_authors.type: user
paper_authors.user: gustavosoares
paper_title: Generative AI for Programming Education: Benchmarking ChatGPT, GPT-4,
    and Human Tutors
paper_summary: Generative AI and large language models hold great promise in enhancing
  computing education by powering next-generation educational technologies for
  introductory programming. Recent works have studied these models for different
  scenarios relevant to programming education; however, these works are limited
  for several reasons, as they typically consider already outdated models or only
  specific scenario(s). Consequently, there is a lack of a systematic study that
  benchmarks state-of-the-art models for a comprehensive set of programming
  education scenarios. In our work, we systematically evaluate two models,
  ChatGPT (based on GPT-3.5) and GPT-4, and compare their performance with human
  tutors for a variety of scenarios. We evaluate using five introductory Python
  programming problems and real-world buggy programs from an online platform, and
  assess performance using expert-based annotations. Our results show that GPT-4
  drastically outperforms ChatGPT (based on GPT-3.5) and comes close to human
  tutors' performance for several scenarios. These results also highlight
  settings where GPT-4 still struggles, providing exciting future directions on
  developing techniques to improve the performance of these models.
paper_ai_summary: Systematic evaluation of ChatGPT and GPT-4 in educational scenarios demonstrates GPT-4's superior performance over ChatGPT and identifies areas needing improvement.
paper_ai_keywords: GPT-3.5 GPT-4 ChatGPT introductory Python programming expert-based annotations