modalities and extend the self-attention mechanism of the transformer architecture to compute their inter-dependencies
in a disentangled manner, as explained in the following section. Furthermore, instead of the traditional left-to-right next
token prediction during self-supervised training, we employ a text infilling objective that better leverages contextual
information.
DOCLLM