  One challenge with open-domain dialogue systems is the need to produce
truthful, high-quality responses on any topic. We aim to improve the quality
and coverage of Athena, an Alexa Prize dialogue system. We experiment with
few-shot prompt-based learning, comparing GPT-Neo to Jurassic-1, for the
movies, music, TV, sports, and video game domains, both within and
cross-domain, with different prompt set sizes (2, 3, 10), formats, and meaning
representations consisting of either sets of WikiData KG triples, or dialogue
acts. Our evaluation uses BLEURT and human metrics, and shows that with 10-shot
prompting, Athena-Jurassic's performance is significantly better for coherence
and semantic accuracy. Experiments with 2-shot cross-domain prompts results in
a huge performance drop for Athena-GPT-Neo, whose semantic accuracy falls to
0.41, and whose untrue hallucination rate increases to 12%. Experiments with
dialogue acts for video games show that with 10-shot prompting, both models
learn to control dialogue acts, but Athena-Jurassic has significantly higher
coherence, and only 4% untrue hallucinations. Our results suggest that
Athena-Jurassic produces high enough quality outputs to be useful in live
systems with real users. To our knowledge, these are the first results
demonstrating that few-shot semantic prompt-based learning can create NLGs that
generalize to new domains, and produce high-quality, semantically-controlled,
conversational responses directly from meaning representations.

Computation and Language