Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. In Proceedings of ICLR.

Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. (2018). Can a suit of armor conduct electricity?

a new dataset for open book question answering. In Proceedings of EMNLP.

7

Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Tazi, N., Piktus, A., Pyysalo, S., Wolf, T., and

Raffel, C. (2023). Scaling data-constrained language models. In Proceedings of NeurIPS.

OpenAI (2023). Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. (2021). Winogrande: An adversarial

winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Shazeer, N. (2020). GLU variants improve transformer. CoRR, abs/2002.05202.

Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. (2023). SlimPajama:

A 627B token cleaned and deduplicated version of RedPajama.
TinyLlama