WatChat: Explaining perplexing programs by debugging mental models
<|reference_start|>WatChat: Explaining perplexing programs by debugging mental models: Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model.<|reference_end|>
@article{chandra2024watchat:,
    title={WatChat: Explaining perplexing programs by debugging mental models},
    author={Kartik Chandra, Katherine M. Collins, Will Crichton, Tony Chen,
  Tzu-Mao Li, Adrian Weller, Rachit Nigam, Joshua Tenenbaum, Jonathan
  Ragan-Kelley},
    journal={arXiv preprint arXiv:2403.05334},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.05334},
    primaryClass={cs.PL cs.AI cs.HC}
}