Vt = HWt,v,

H′ = sof tmax(

At√
d

)Vt.

(2)

In DocLLM, the input is represented as x = {(xi, bi)}T
i=1, where bi = (left, top, right, bottom) is the bounding
box corresponding to xi. To capture the new modality (i.e. spatial information), we encode the bounding boxes into
hidden vectors represented by S ∈ RT×d. We then decompose the attention matrix computation into four different
scores, namely text-to-text, text-to-spatial, spatial-to-text and spatial-to-spatial. Formally, the new attention mechanism
is calculated as:

,

⊺

⊺

⊺

iKt
j

iKs
j

i Kt
j

i Ks
j

+ λt,sQt

+ λs,tQs

+ λs,sQs

Ks = SWs,k

Qs = SWs,q,
⊺
Ai,j = Qt
DOCLLM