The existence of formal linguistic abilities in
PLMs has been well established for some time now
(Rogers et al., 2020). In fact, it has been shown
that pre-training on as few as between 10 and 100
million tokens is sufficient for PLMs to capture
a significant amount of formal linguistic informa-
tion (Zhang et al., 2021). It has been argued that
such linguistic information might essentially be
theoretical constructs, a notion that gains traction
especially given that their existence does not pre-
cede the use of language (Tayyar Madabushi et al.,
2023). Given that it is possible that linguistic ele-
ments are statistical generalisations that arise out
of language use, representing intricate patterns that
are necessitated by various goals tied to efficient
communication, it is not surprising that LLMs have
access to such information when trained on large
corpora. Even more unsurprising is the fact that
LLMs, when trained more extensively, also capture

5
AreEmergentAbilitiesinLarge Language Models just In-Context