  Many recent efforts aim to augment language models with relevant information
retrieved from a database at test time. We avoid the need for prompt
engineering by directly fine-tuning the model on data retrieved at test time
using its standard training setup. For this purpose, we build a large-scale
distributed nearest neighbor index based on text embeddings of the Pile
dataset. Given a query to a language model, our system retrieves the neighbors
of the query and fine-tunes the model on the text data corresponding to those
neighbors. Surprisingly, retrieving and training on as few as 20 neighbors,
each for only one gradient iteration, drastically improves performance across
more than twenty language modeling tasks in the Pile benchmark. For example,
test-time training significantly narrows the performance gap between a small
GPT2 model and a GPTNeo model, more than ten times larger, that was
specifically trained to convergence on the Pile. Sufficient index quality and
size, however, are important. Our work establishes a valuable first baseline
for implementing test-time training in the context of large language models,
opening the door to numerous promising research avenues.

Computation and Language;Machine Learning