Baselines. We compare our model to numerous popular baselines, all of which are generative.
Speciﬁcally, we consider RAG (Lewis et al., 2020), Retro (Borgeaud et al., 2021), EMDR2 (Singh
et al., 2021) and FiD/FiD-Distill Izacard & Grave (2020a;b). For fair comparison, we differentiate
models that use DPR for retrieval from those that leverage stronger ones.

Ablations. To help us to understand the contribution of the re-ranking module, we ran the same
experiment when greedily packing passages into the context window of the frozen LM based on the
original retriever relevance scores, which are computed based on separate dense representations of
the question and passage.

3.2 RESULTS
STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS