  Voice dictation is an increasingly important text input modality. Existing
systems that allow both dictation and editing-by-voice restrict their command
language to flat templates invoked by trigger words. In this work, we study the
feasibility of allowing users to interrupt their dictation with spoken editing
commands in open-ended natural language. We introduce a new task and dataset,
TERTiUS, to experiment with such systems. To support this flexibility in
real-time, a system must incrementally segment and classify spans of speech as
either dictation or command, and interpret the spans that are commands. We
experiment with using large pre-trained language models to predict the edited
text, or alternatively, to predict a small text-editing program. Experiments
show a natural trade-off between model accuracy and latency: a smaller model
achieves 30% end-state accuracy with 1.3 seconds of latency, while a larger
model achieves 55% end-state accuracy with 7 seconds of latency.

Computation and Language