InstructRetro is an autoregressive decoder-only language model (LM) with retrieval-augmented pretraining and instruction tuning.