Gl\'orIA -- A Generative and Open Large Language Model for Portuguese
<|reference_start|>Gl\'orIA -- A Generative and Open Large Language Model for Portuguese: Significant strides have been made in natural language tasks, largely attributed to the emergence of powerful large language models (LLMs). These models, pre-trained on extensive and diverse corpora, have become increasingly capable of comprehending the intricacies of language. Despite the abundance of LLMs for many high-resource languages, the availability of such models remains limited for European Portuguese. We introduce Gl\'orIA, a robust European Portuguese decoder LLM. To pre-train Gl\'orIA, we assembled a comprehensive PT-PT text corpus comprising 35 billion tokens from various sources. We present our pre-training methodology, followed by an assessment of the model's effectiveness on multiple downstream tasks. Additionally, to evaluate our models' language modeling capabilities, we introduce CALAME-PT (Context-Aware LAnguage Modeling Evaluation for Portuguese), the first Portuguese zero-shot language-modeling benchmark. Evaluation shows that Gl\'orIA significantly outperforms existing open PT decoder models in language modeling and that it can generate sound, knowledge-rich, and coherent PT-PT text. The model also exhibits strong potential for various downstream tasks.<|reference_end|>
@article{lopes2024gl\'oria,
    title={Gl\'orIA -- A Generative and Open Large Language Model for Portuguese},
    author={Ricardo Lopes and Jo~ao Magalh~aes and David Semedo},
    journal={arXiv preprint arXiv:2402.12969},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.12969},
    primaryClass={cs.CL cs.AI}
}