5 Ablation Studies

We conduct ablation studies to validate the three contributions of DocLLM: (1) disentangled spatial features, (2) the
block infilling pre-training objective, and (3) the masking strategy used for decoding.
For all ablations, we use Next Token Prediction (NTP) out-of-sample accuracy to compare configurations at the
pre-training stage. Due to resource restrictions, each experiment uses a subset of our pre-training corpus: we randomly
sample 100,000 chunks and predict on 1,000 unseen documents. A chunk is a pack of documents concatenated one by
one with the total length less than maximum input length. The hyperparameters are set consistently following Table 4
across all ablation experiments.

5This is done to remain consistent with the results reported by other SotA models.

9

Table 6: Performance comparison on three held-out VRDU datasets in the STDD setting against non-multimodal LLMs.

Model

Size
GPT-4+OCR ∼1T
Llama2+OCR
7B
1B
DocLLM-1B
DocLLM-7B
7B
DOCLLM