Generation-based Open-QA An emerging alternative
approach to Open-QA is to model it as a sequence predic-
tion task: simply encode the question, and then decode the
answer token-by-token based on the encoding. While it was
initially unclear how large amounts of knowledge could be
injected into the model, GPT-2 (Radford et al., 2019) hinted
at the possibility of directly generating answers without us-
ing any given context via sequence-to-sequence. However,
their performance was not competitive possibly due to the
lack of ﬁne-tuning. Orthogonally, T5 (Raffel et al., 2019)
showed that directly generating answers without explicit
extraction from the given context is viable approach, but
they only experimented on the reading comprehension task,
where a context document is provided.
For the most competitive and comparable generation-based
baseline, we compare to concurrent work which ﬁne-tunes
T5 for Open-QA (Roberts et al., 2020).4 We compare
REALM