In this paper, we introduced DocLLM, a lightweight extension to traditional large language models, tailored for generative
reasoning over documents with rich layouts. Unlike existing multimodal LLMs, DocLLM strategically omits costly
image encoders, instead prioritizing bounding box information to effectively capture the spatial layout structure of
documents. This is achieved through a disentangled attention approach, decomposing the attention mechanism in
classical transformers, and enhancing with cross-alignment between text and spatial modalities in structured documents.
Notably, our model addresses the challenges posed by irregular layouts and heterogeneous content by employing
a pre-training objective that focuses on learning to infill block texts. We fine-tuned the pre-trained model using a
comprehensive instruction dataset. Our evaluation across various document intelligence tasks demonstrates that DocLLM
DOCLLM