but it was
late.
too
Lily
was
sad
and
angry.

and Lily’s
mom
opened it.
”Ben, what
are
you
doing?”
Lily asked.

but no one
answered.
He knocked
again, but
still no one
answered.

but no one
answered.
He knocked
again and
again, but
still no one
came.

but no one
answered.
Lily
sad
lonely.

felt
and

but
did
answer.

Lily
not

but no one
came to an-
swer.

Figure 10: Performance of different models on reasoning prompts

She
no.
to
wants
at
stay
home
and
play with
her toys.

at

both
reached for
it
the
same time.
Father
said,
you
share
cake,
will
to
you.”

”If
don’t
the
I
have
punish

but no one
answered.
He tried to
open
the
door, but it
was locked.

14

GPT2-XL
(1.5B)

”I
think
she’s up in
the
trees,
the
by
lake,”
she
replied.

Brown

a
dog

The girl re-
sponded, ”I
have pieces
of candy.”

2.5M
8 layers

8.3M
8 layers

28M
8 layers

33M
4 layers

21M
1 layer

33M
2 layers

Prompt

”Hi Jane, have you seen
Alice? I can’t find her
anywhere”, said Jack.

1M
8 layers
TinyStories-HowSmallCanLanguageModelsBeandStillSpeak CoherentEnglish?