title: Textbooks Are All You Need
summary: We introduce phi-1, a new large language model for code, with significantly
  smaller size than competing models: phi-1 is a Transformer-based model with
  1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
  quality" data from the web (6B tokens) and synthetically generated textbooks
  and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
  pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
  surprising emergent properties compared to phi-1-base, our model before our
  finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
  model with 350M parameters trained with the same pipeline as phi-1 that still
  achieves 45% on HumanEval.
paper_authors._id: 64926f44ef19676db9353f94
paper_authors.name: Suriya Gunasekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:18:37.285Z
paper_authors._id: 63e1b4f77fbb6ae4d4f36aa4
paper_authors.avatarUrl: /avatars/7febc332d393e0bb94255104527ffe2a.svg
paper_authors.fullname: Gunasekar
paper_authors.type: user
paper_authors.user: suriyagunasekar
paper_authors._id: 64926f44ef19676db9353f95
paper_authors.name: Yi Zhang
paper_authors._id: 64926f44ef19676db9353f96
paper_authors.name: Jyoti Aneja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:19:58.085Z
paper_authors._id: 623e17038972a8c030af23fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623e17038972a8c030af23fa/2ERyhCNAdV2wt4uRy3Wq5.jpeg
paper_authors.fullname: Jyoti Aneja
paper_authors.type: user
paper_authors.user: jyotiA
paper_authors._id: 64926f44ef19676db9353f97
paper_authors.name: Caio César Teodoro Mendes
paper_authors._id: 64926f44ef19676db9353f98
paper_authors.name: Allie Del Giorno
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:20:21.299Z
paper_authors._id: 63f5562471a5d395c721cd8e
paper_authors.avatarUrl: /avatars/ac296b6017e86ea04c73803fe2c44433.svg
paper_authors.fullname: Allie Del Giorno
paper_authors.type: user
paper_authors.user: microallie
paper_authors._id: 64926f44ef19676db9353f99
paper_authors.name: Sivakanth Gopi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:20:47.585Z
paper_authors._id: 644c401926de083beb98544e
paper_authors.avatarUrl: /avatars/76dfef0ea98006cd2681b4de3aca9d22.svg
paper_authors.fullname: Sivakanth Gopi
paper_authors.type: user
paper_authors.user: Sivakanth
paper_authors._id: 64926f44ef19676db9353f9a
paper_authors.name: Mojan Javaheripi
paper_authors._id: 64926f44ef19676db9353f9b
paper_authors.name: Piero Kauffmann
paper_authors._id: 64926f44ef19676db9353f9c
paper_authors.name: Gustavo de Rosa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:23:05.446Z
paper_authors._id: 6157454831624da88210e627
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666203761402-6157454831624da88210e627.jpeg
paper_authors.fullname: Gustavo de Rosa
paper_authors.type: user
paper_authors.user: gugarosa
paper_authors._id: 64926f44ef19676db9353f9d
paper_authors.name: Olli Saarikivi
paper_authors._id: 64926f44ef19676db9353f9e
paper_authors.name: Adil Salim
paper_authors._id: 64926f44ef19676db9353f9f
paper_authors.name: Shital Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:23:43.003Z
paper_authors._id: 5fea740dbb3114fa14c82934
paper_authors.avatarUrl: /avatars/5f9fe9aaf74a3b930b6fcebcf8a4e4cc.svg
paper_authors.fullname: Shital Shah
paper_authors.type: user
paper_authors.user: sytelus
paper_authors._id: 64926f44ef19676db9353fa0
paper_authors.name: Harkirat Singh Behl
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:23:57.897Z
paper_authors._id: 63c6aa9e656e7822e2359d9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
paper_authors.fullname: Harkirat Behl
paper_authors.type: user
paper_authors.user: Harkirat
paper_authors._id: 64926f44ef19676db9353fa1
paper_authors.name: Xin Wang
paper_authors._id: 64926f44ef19676db9353fa2
paper_authors.name: Sébastien Bubeck
paper_authors._id: 64926f44ef19676db9353fa3
paper_authors.name: Ronen Eldan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:24:33.710Z
paper_authors._id: 633723a80267ebcf0264c06b
paper_authors.avatarUrl: /avatars/22bb971597e9f3abfa343280a9d0f65f.svg
paper_authors.fullname: Ronen Eldan
paper_authors.type: user
paper_authors.user: roneneldan
paper_authors._id: 64926f44ef19676db9353fa4
paper_authors.name: Adam Tauman Kalai
paper_authors._id: 64926f44ef19676db9353fa5
paper_authors.name: Yin Tat Lee
paper_authors._id: 64926f44ef19676db9353fa6
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-21T09:25:09.590Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_title: Textbooks Are All You Need
paper_summary: We introduce phi-1, a new large language model for code, with significantly
  smaller size than competing models: phi-1 is a Transformer-based model with
  1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook
  quality" data from the web (6B tokens) and synthetically generated textbooks
  and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains
  pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays
  surprising emergent properties compared to phi-1-base, our model before our
  finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller
  model with 350M parameters trained with the same pipeline as phi-1 that still
  achieves 45% on HumanEval.
paper_ai_keywords: Transformer-based model pass@1 accuracy HumanEval MBPP