  Scholastic trivia competitions test knowledge and intelligence through
mastery of question answering. Modern question answering benchmarks are one
variant of the Turing test. Specifically, answering a set of questions as well
as a human is a minimum bar towards demonstrating human-like intelligence. This
paper makes the case that the format of one competition -- where participants
can answer in the middle of hearing a question (incremental) -- better
differentiates the skill between (human or machine) players. Additionally,
merging a sequential decision-making sub-task with question answering (QA)
provides a good setting for research in model calibration and opponent
modeling. Thus, embedded in this task are three machine learning challenges:
(1) factoid QA over thousands of Wikipedia-like answers, (2) calibration of the
QA model's confidence scores, and (3) sequential decision-making that
incorporates knowledge of the QA model, its calibration, and what the opponent
may do. We make two contributions: (1) collecting and curating a large factoid
QA dataset and an accompanying gameplay dataset, and (2) developing a model
that addresses these three machine learning challenges. In addition to offline
evaluation, we pitted our model against some of the most accomplished trivia
players in the world in a series of exhibition matches spanning several years.
Throughout this paper, we show that collaborations with the vibrant trivia
community have contributed to the quality of our dataset, spawned new research
directions, and doubled as an exciting way to engage the public with research
in machine learning and natural language processing.

Computation and Language