program (Devlin et al., 2017), or even generate programmatic policies in reinforcement learning
settings (Trivedi et al., 2021).
Automatic code completion is also relevant to our work and has become an integral part of most code
editors and integrated development environments (IDEs). While typing, these tools suggest possible
continuations, greatly improving programming productivity. The earliest code completion systems
were purely syntax-based. Hindle et al. (2012) provided empirical evidence that code can be modeled
by statistical 𝑛-gram language models, and capitalised on this to develop a simple code completion
engine for Java. More recent intelligent code completion systems can learn from history (Robbes and
alphacode