  Physics education researchers are interested in using the tools of machine
learning and natural language processing to make quantitative claims from
natural language and text data, such as open-ended responses to survey
questions. The aspiration is that this form of machine coding may be more
efficient and consistent than human coding, allowing much larger and broader
data sets to be analyzed than is practical with human coders. Existing work
that uses these tools, however, does not investigate norms that allow for
trustworthy quantitative claims without full reliance on cross-checking with
human coding, which defeats the purpose of using these automated tools. Here we
propose a four-part method for making such claims with supervised natural
language processing: evaluating a trained model, calculating statistical
uncertainty, calculating systematic uncertainty from the trained algorithm, and
calculating systematic uncertainty from novel data sources. We provide evidence
for this method using data from two distinct short response survey questions
with two distinct coding schemes. We also provide a real-world example of using
these practices to machine code a data set unseen by human coders. We offer
recommendations to guide physics education researchers who may use
machine-coding methods in the future.

Physics Education