Re-ranker Training. We trained our re-ranker following the same protocol used by Karpukhin
et al. (2020) to train their extractive reader. We based the re-ranker architecture on the 110M
parameter BERT-base (Devlin et al., 2019) model, such that a forward pass through the re-ranker
incurs negligible run-time cost relative to a single pass through our 7B/17B parameter frozen LMs.
During training, we sampled one positive and 23 negative passages from the top 100 passages
returned by the retrieval system for each question. The training objective was to maximize the
marginal log-likelihood of the start and end of all the correct answer spans in the positive passage
(the answer string may appear multiple times in one passage), combined with the log-likelihood of
the positive passage being selected. We used a batch size of 16, and trained the re-ranker for up to
30K steps with a learning rate of 1 · 10−5 using Adam (Kingma & Ba, 2014), linear scheduling with
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS