joinBERT(x) = [CLS]x[SEP]

joinBERT(x1, x2) = [CLS]x1[SEP]x2[SEP]

As in Devlin et al. (2018), we pass this into a Transformer,
which produces one vector for each token, including the
vector corresponding to [CLS] which is used as a “pooled”
representation of the sequence (denoted BERTCLS). Finally,
we perform a linear projection to reduce the dimensionality
of the vector, denoted as a projection matrix W:

Embedinput(x) = WinputBERTCLS(joinBERT(x))

Embeddoc(z) = WdocBERTCLS(joinBERT(ztitle, zbody))

where ztitle is the document’s title and zbody is its body. We
let θ denote all parameters associated with the retriever,
which include the Transformer and projection matrices.
REALM