  Qualitative methods like interviews produce richer data in comparison with
quantitative surveys, but are difficult to scale. Switching from web-based
questionnaires to interactive chatbots offers a compromise, improving user
engagement and response quality. Uptake remains limited, however, because of
differences in users' expectations versus the capabilities of natural language
processing methods. In this study, we evaluate the potential of large language
models (LLMs) to support an information elicitation chatbot that narrows this
"gulf of expectations" (Luger & Sellen 2016). We conduct a user study in which
participants (N = 399) were randomly assigned to interact with a rule-based
chatbot versus one of two LLM-augmented chatbots. We observe limited evidence
of differences in user engagement or response richness between conditions.
However, the addition of LLM-based dynamic probing skills produces significant
improvements in both quantitative and qualitative measures of user experience,
consistent with a narrowing of the expectations gulf.

Human-Computer Interaction