3.3.3 Manual Evaluation of Responses
To further mitigate the possibility that models
which are not instruction-tuned are not unfairly

11

penalised, we perform a manual analysis of their
responses. This is required, due to the possibility
that these models generate the correct answer, al-
beit in a format that is not semantically most similar
to the option provided. To account for this possi-
bility, we preform a manual post-hoc analysis of a
subset of the outputs generated by models which
are not instruction-tuned.

3.4 Experimental Setup

Family

Tasks

Model
GPT-2

GPT-2-IT
GPT-2-XL

GPT-2-XL-IT

GPT-J
GPT-JT
davinci

text-davinci-001
text-davinci-003

T5-small

FLAN-T5-small

T5-large

FLAN-T5-large

Falcon-7B

Falcon-7B-IT
Falcon-40B

Falcon-40B-IT

LLaMA-7B
LLaMA-13B
LLaMA-30B

GPT

T5

Falcon

LLaMA

All 22 Tasks

Logical Deductions,
Social IQA, GSM8K,
Tracking Shuffled
Objects
AreEmergentAbilitiesinLarge Language Models just In-Context