context of models with a substantial parameter
count of 175B (Wei et al., 2022a). We leverage
this phenomenon by using the “adversarial prompt
setting” (Section 3.3.1), wherein the model is re-
quired to generate output choices, such as options
"a" or "b," instead of the target choice. In this set-
ting we evaluate models using a relaxed version of
exact match wherein an answer is marked correct if
it contains the correct target option. This flexibility
is once again designed to allow us to detect any
possible indication of emergence. Note that this
assessment allows us to circumvent the necessity
for employing less precise evaluation criteria as is
required when evaluating more verbose responses.
The results of this evaluation on the seven of 22
tasks wherein the performance is above the random
baseline are presented in Figure 5.

Of

these seven tasks on which the non-
AreEmergentAbilitiesinLarge Language Models just In-Context