Textbooks Are All You Need II: phi-1.5 technical report
We continue the investigation into the power of smaller Transformer-based
language models as initiated by TinyStories -- a 10 million parameter
model that can produce coherent English -- and the follow-up work on
phi-1, a 1.3 billion parameter model with Python coding performance
close to the state-of-the-art. The latter work proposed to use existing Large
Language Models (LLMs) to generate ``textbook quality" data as a way to enhance
the learning process compared to traditional web data. We follow the
``Textbooks Are All You Need" approach, focusing this time on common sense
reasoning in natural language, and create a new 1.3 billion parameter model
named phi-1.5, with performance on natural language tasks comparable
to models 5x larger, and surpassing most non-frontier LLMs on more complex
reasoning tasks such as grade-school mathematics and basic coding. More
generally, phi-1.5 exhibits many of the traits of much larger LLMs,
both good -- such as the ability to ``think step by step" or perform some
rudimentary in-context learning -- and bad, including hallucinations and the
potential for toxic and biased generations -- encouragingly though, we are
seeing improvement on that front thanks to the absence of web data. We
open-source phi-1.5 to promote further research on these urgent
topics.
64ffd61085a884a964aec59a
Yuanzhi Li
admin_assigned
2023-09-12T10:58:57.017Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
64ffd61085a884a964aec59b
Sébastien Bubeck
admin_assigned
2023-09-12T11:00:18.756Z
64557408c9c0dcc8c24a7a92
/avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
Sebastien Bubeck
user
sebubeck
64ffd61085a884a964aec59c
Ronen Eldan
admin_assigned
2023-09-12T10:59:17.424Z
633723a80267ebcf0264c06b
/avatars/22bb971597e9f3abfa343280a9d0f65f.svg
Ronen Eldan
user
roneneldan
64ffd61085a884a964aec59d
Allie Del Giorno
admin_assigned
2023-09-12T10:59:39.427Z
63f5562471a5d395c721cd8e
/avatars/ac296b6017e86ea04c73803fe2c44433.svg
Allie Del Giorno
user
microallie
64ffd61085a884a964aec59e
Suriya Gunasekar
admin_assigned
2023-09-12T10:59:54.922Z
63e1b4f77fbb6ae4d4f36aa4
/avatars/7febc332d393e0bb94255104527ffe2a.svg
Gunasekar
user
suriyagunasekar
64ffd61085a884a964aec59f
Yin Tat Lee
admin_assigned
2023-09-12T11:00:03.159Z
6303297604c75db08b8972e2
/avatars/693b2dada2244ba5d97df9518f473ccb.svg
Yin Tat Lee
user
yintat
Textbooks Are All You Need II: phi-1.5 technical report
We continue the investigation into the power of smaller Transformer-based
language models as initiated by TinyStories -- a 10 million parameter
model that can produce coherent English -- and the follow-up work on
phi-1, a 1.3 billion parameter model with Python coding performance
close to the state-of-the-art. The latter work proposed to use existing Large
Language Models (LLMs) to generate ``textbook quality" data as a way to enhance
the learning process compared to traditional web data. We follow the
``Textbooks Are All You Need" approach, focusing this time on common sense
reasoning in natural language, and create a new 1.3 billion parameter model
named phi-1.5, with performance on natural language tasks comparable
to models 5x larger, and surpassing most non-frontier LLMs on more complex
reasoning tasks such as grade-school mathematics and basic coding. More
generally, phi-1.5 exhibits many of the traits of much larger LLMs,
both good -- such as the ability to ``think step by step" or perform some
rudimentary in-context learning -- and bad, including hallucinations and the
potential for toxic and biased generations -- encouragingly though, we are
seeing improvement on that front thanks to the absence of web data. We
open-source phi-1.5 to promote further research on these urgent
topics.
A new 1.3 billion parameter Transformer-based language model, phi-1.5, demonstrates comparable performance to much larger models on common sense reasoning and complex tasks despite the absence of web data.
Transformer-based language models TinyStories phi-1 Large Language Models (LLMs) textbook quality data natural language tasks common sense reasoning grade-school mathematics basic coding in-context learning hallucinations toxic and biased generations