title: Textbooks Are All You Need II: phi-1.5 technical report
summary: We continue the investigation into the power of smaller Transformer-based
  language models as initiated by TinyStories -- a 10 million parameter
  model that can produce coherent English -- and the follow-up work on
  phi-1, a 1.3 billion parameter model with Python coding performance
  close to the state-of-the-art. The latter work proposed to use existing Large
  Language Models (LLMs) to generate ``textbook quality" data as a way to enhance
  the learning process compared to traditional web data. We follow the
  ``Textbooks Are All You Need" approach, focusing this time on common sense
  reasoning in natural language, and create a new 1.3 billion parameter model
  named phi-1.5, with performance on natural language tasks comparable
  to models 5x larger, and surpassing most non-frontier LLMs on more complex
  reasoning tasks such as grade-school mathematics and basic coding. More
  generally, phi-1.5 exhibits many of the traits of much larger LLMs,
  both good -- such as the ability to ``think step by step" or perform some
  rudimentary in-context learning -- and bad, including hallucinations and the
  potential for toxic and biased generations -- encouragingly though, we are
  seeing improvement on that front thanks to the absence of web data. We
  open-source phi-1.5 to promote further research on these urgent
  topics.
paper_authors._id: 64ffd61085a884a964aec59a
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T10:58:57.017Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 64ffd61085a884a964aec59b
paper_authors.name: Sébastien Bubeck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:00:18.756Z
paper_authors._id: 64557408c9c0dcc8c24a7a92
paper_authors.avatarUrl: /avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
paper_authors.fullname: Sebastien Bubeck
paper_authors.type: user
paper_authors.user: sebubeck
paper_authors._id: 64ffd61085a884a964aec59c
paper_authors.name: Ronen Eldan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T10:59:17.424Z
paper_authors._id: 633723a80267ebcf0264c06b
paper_authors.avatarUrl: /avatars/22bb971597e9f3abfa343280a9d0f65f.svg
paper_authors.fullname: Ronen Eldan
paper_authors.type: user
paper_authors.user: roneneldan
paper_authors._id: 64ffd61085a884a964aec59d
paper_authors.name: Allie Del Giorno
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T10:59:39.427Z
paper_authors._id: 63f5562471a5d395c721cd8e
paper_authors.avatarUrl: /avatars/ac296b6017e86ea04c73803fe2c44433.svg
paper_authors.fullname: Allie Del Giorno
paper_authors.type: user
paper_authors.user: microallie
paper_authors._id: 64ffd61085a884a964aec59e
paper_authors.name: Suriya Gunasekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T10:59:54.922Z
paper_authors._id: 63e1b4f77fbb6ae4d4f36aa4
paper_authors.avatarUrl: /avatars/7febc332d393e0bb94255104527ffe2a.svg
paper_authors.fullname: Gunasekar
paper_authors.type: user
paper_authors.user: suriyagunasekar
paper_authors._id: 64ffd61085a884a964aec59f
paper_authors.name: Yin Tat Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:00:03.159Z
paper_authors._id: 6303297604c75db08b8972e2
paper_authors.avatarUrl: /avatars/693b2dada2244ba5d97df9518f473ccb.svg
paper_authors.fullname: Yin Tat Lee
paper_authors.type: user
paper_authors.user: yintat
paper_title: Textbooks Are All You Need II: phi-1.5 technical report
paper_summary: We continue the investigation into the power of smaller Transformer-based
  language models as initiated by TinyStories -- a 10 million parameter
  model that can produce coherent English -- and the follow-up work on
  phi-1, a 1.3 billion parameter model with Python coding performance
  close to the state-of-the-art. The latter work proposed to use existing Large
  Language Models (LLMs) to generate ``textbook quality" data as a way to enhance
  the learning process compared to traditional web data. We follow the
  ``Textbooks Are All You Need" approach, focusing this time on common sense
  reasoning in natural language, and create a new 1.3 billion parameter model
  named phi-1.5, with performance on natural language tasks comparable
  to models 5x larger, and surpassing most non-frontier LLMs on more complex
  reasoning tasks such as grade-school mathematics and basic coding. More
  generally, phi-1.5 exhibits many of the traits of much larger LLMs,
  both good -- such as the ability to ``think step by step" or perform some
  rudimentary in-context learning -- and bad, including hallucinations and the
  potential for toxic and biased generations -- encouragingly though, we are
  seeing improvement on that front thanks to the absence of web data. We
  open-source phi-1.5 to promote further research on these urgent
  topics.
paper_ai_summary: A new 1.3 billion parameter Transformer-based language model, phi-1.5, demonstrates comparable performance to much larger models on common sense reasoning and complex tasks despite the absence of web data.
paper_ai_keywords: Transformer-based language models TinyStories phi-1 Large Language Models (LLMs) textbook quality data natural language tasks common sense reasoning grade-school mathematics basic coding in-context learning hallucinations toxic and biased generations