of language models across various categories: a)
tasks that are considered emergent for GPT-3, b)
those considered not to be emergent despite LLMs’
ability to solve them as this is predictable based
on the success of smaller models, and c) tasks on
which LLMs completely fail. GSM8K, and one
of the emergent tasks from BIG-Bench, Modified
Arithmetic, as originally structured, do not adopt
a multiple-choice format. However, for alignment
with our other experiments, we adapt them into a
multiple-choice setup by including numeric distrac-
tors as follows: To ensure adversarial nature, we
automatically select numbers in proximity to and
randomly distant from the correct answer.
AreEmergentAbilitiesinLarge Language Models just In-Context