Characterizing emergent phenomena in large language models
&gt; we discuss the phenomena of emergent abilities, which we define as abilities that are not present in small models but are present in larger models<p>Reading anything by major researchers in AI feels like an adversarial battle where they&#x27;re trying to misuse as much technical scientific and philosophical language as possible and we adjacent are trying to hold the line.<p>In philosophy and esp. the philosophy of science, emergence is a relation between a whole and its parts such that a property of the whole does not obtain just in virtue of properties of its parts taken in isolation. &quot;Emergence&quot; has this prior positive, semi-magical, scientific association which confuses the issue in this case.<p>No properties of the LLM obtain from its parts differently as parameters scale, the mechanism is the same. The performance differs not due to emergence, but due to the &quot;modelling gap&quot; present between the statistical structure of free text and that of mathematics. With enough examples, the  gap closes... indeed, you can model the addition function (add(x, y) = x + y) just by an infinite sample of its domains.<p>A better technical term here might be &quot;scale-dependent capabilities&quot;. For LLM, simple arithmetic is extremely scale dependent, whereas basic text generation is less-so. The reason for this seems obvious, as given above... so the use of the term &quot;emergence&quot; here I interpert as more PRish mystification.