title: Phi-4 Technical Report
summary: We present phi-4, a 14-billion parameter language model developed with a
  training recipe that is centrally focused on data quality. Unlike most language
  models, where pre-training is based primarily on organic data sources such as
  web content or code, phi-4 strategically incorporates synthetic data throughout
  the training process. While previous models in the Phi family largely distill
  the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
  surpasses its teacher model on STEM-focused QA capabilities, giving evidence
  that our data-generation and post-training techniques go beyond distillation.
  Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
  performance relative to its size -- especially on reasoning-focused benchmarks
  -- due to improved data, training curriculum, and innovations in the
  post-training scheme.
paper_authors._id: 675b8e0dcda0ebd0345d6ffa
paper_authors.name: Marah Abdin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:10.073Z
paper_authors._id: 649af1e3ca20306aeef590b8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649af1e3ca20306aeef590b8/Tvy8yf7zDO2FoNs03tWj6.jpeg
paper_authors.fullname: Marah Abdin
paper_authors.type: user
paper_authors.user: marah-abdin
paper_authors._id: 675b8e0dcda0ebd0345d6ffb
paper_authors.name: Jyoti Aneja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:16.014Z
paper_authors._id: 623e17038972a8c030af23fa
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623e17038972a8c030af23fa/2ERyhCNAdV2wt4uRy3Wq5.jpeg
paper_authors.fullname: Jyoti Aneja
paper_authors.type: user
paper_authors.user: jyotiA
paper_authors._id: 675b8e0dcda0ebd0345d6ffc
paper_authors.name: Harkirat Behl
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:21.406Z
paper_authors._id: 63c6aa9e656e7822e2359d9c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
paper_authors.fullname: Harkirat Behl
paper_authors.type: user
paper_authors.user: Harkirat
paper_authors._id: 675b8e0dcda0ebd0345d6ffd
paper_authors.name: Sébastien Bubeck
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:27.076Z
paper_authors._id: 64557408c9c0dcc8c24a7a92
paper_authors.avatarUrl: /avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
paper_authors.fullname: Sebastien Bubeck
paper_authors.type: user
paper_authors.user: sebubeck
paper_authors._id: 675b8e0dcda0ebd0345d6ffe
paper_authors.name: Ronen Eldan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:32.923Z
paper_authors._id: 633723a80267ebcf0264c06b
paper_authors.avatarUrl: /avatars/22bb971597e9f3abfa343280a9d0f65f.svg
paper_authors.fullname: Ronen Eldan
paper_authors.type: user
paper_authors.user: roneneldan
paper_authors._id: 675b8e0dcda0ebd0345d6fff
paper_authors.name: Suriya Gunasekar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:34:38.249Z
paper_authors._id: 63e1b4f77fbb6ae4d4f36aa4
paper_authors.avatarUrl: /avatars/7febc332d393e0bb94255104527ffe2a.svg
paper_authors.fullname: Gunasekar
paper_authors.type: user
paper_authors.user: suriyagunasekar
paper_authors._id: 675b8e0dcda0ebd0345d7000
paper_authors.name: Michael Harrison
paper_authors._id: 675b8e0dcda0ebd0345d7001
paper_authors.name: Russell J. Hewett
paper_authors._id: 675b8e0dcda0ebd0345d7002
paper_authors.name: Mojan Javaheripi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:35:16.168Z
paper_authors._id: 643dbfa31e5be78c66440fcf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/643dbfa31e5be78c66440fcf/VtLrAlPY_4ncVrM4eL7V-.jpeg
paper_authors.fullname: Mojan Javaheripi
paper_authors.type: user
paper_authors.user: mojanjp
paper_authors._id: 675b8e0dcda0ebd0345d7003
paper_authors.name: Piero Kauffmann
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:35:24.374Z
paper_authors._id: 662699f051cedbbb0b1a9e81
paper_authors.avatarUrl: /avatars/8d29d2a7b9c7d714617e3961f204bfc6.svg
paper_authors.fullname: Piero Kauffmann
paper_authors.type: user
paper_authors.user: piero2c
paper_authors._id: 675b8e0dcda0ebd0345d7004
paper_authors.name: James R. Lee
paper_authors._id: 675b8e0dcda0ebd0345d7005
paper_authors.name: Yin Tat Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:35:58.047Z
paper_authors._id: 6303297604c75db08b8972e2
paper_authors.avatarUrl: /avatars/693b2dada2244ba5d97df9518f473ccb.svg
paper_authors.fullname: Yin Tat Lee
paper_authors.type: user
paper_authors.user: yintat
paper_authors._id: 675b8e0dcda0ebd0345d7006
paper_authors.name: Yuanzhi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:36:18.888Z
paper_authors._id: 641d5a313f04f9bf2da17e8c
paper_authors.avatarUrl: /avatars/4035f73c69ddb045573ad96db4f13f04.svg
paper_authors.fullname: Yuanzhi Li
paper_authors.type: user
paper_authors.user: Uushizhu1234
paper_authors._id: 675b8e0dcda0ebd0345d7007
paper_authors.name: Weishung Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:36:27.196Z
paper_authors._id: 637c3ac41c13d2970bf67625
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637c3ac41c13d2970bf67625/K_KEPoWjj4k7EndHeU2Jt.png
paper_authors.fullname: Weishung Liu 
paper_authors.type: user
paper_authors.user: weish
paper_authors._id: 675b8e0dcda0ebd0345d7008
paper_authors.name: Caio C. T. Mendes
paper_authors._id: 675b8e0dcda0ebd0345d7009
paper_authors.name: Anh Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:36:50.168Z
paper_authors._id: 649bc84833486cdd77c01c66
paper_authors.avatarUrl: /avatars/36f4e4bb15c337c4391bfbd234051f4c.svg
paper_authors.fullname: Nguyen Anh
paper_authors.type: user
paper_authors.user: Anhnguyen
paper_authors._id: 675b8e0dcda0ebd0345d700a
paper_authors.name: Eric Price
paper_authors._id: 675b8e0dcda0ebd0345d700b
paper_authors.name: Gustavo de Rosa
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T16:05:51.994Z
paper_authors._id: 6157454831624da88210e627
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666203761402-6157454831624da88210e627.jpeg
paper_authors.fullname: Gustavo de Rosa
paper_authors.type: user
paper_authors.user: gugarosa
paper_authors._id: 675b8e0dcda0ebd0345d700c
paper_authors.name: Olli Saarikivi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:37:26.568Z
paper_authors._id: 6477a7108ab7e732b6d86701
paper_authors.avatarUrl: /avatars/3319db398ce8142bd7d61de1d2cd1b4b.svg
paper_authors.fullname: Olli Saarikivi
paper_authors.type: user
paper_authors.user: olsaarik
paper_authors._id: 675b8e0dcda0ebd0345d700d
paper_authors.name: Adil Salim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:37:31.780Z
paper_authors._id: 670851dc39406c0d7e47c373
paper_authors.avatarUrl: /avatars/f4bff694d2ce4665ad281ae5f5c3346a.svg
paper_authors.fullname: Adil Salim
paper_authors.type: user
paper_authors.user: AdilSlm
paper_authors._id: 675b8e0dcda0ebd0345d700e
paper_authors.name: Shital Shah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:38:03.800Z
paper_authors._id: 5fea740dbb3114fa14c82934
paper_authors.avatarUrl: /avatars/5f9fe9aaf74a3b930b6fcebcf8a4e4cc.svg
paper_authors.fullname: Shital Shah
paper_authors.type: user
paper_authors.user: sytelus
paper_authors._id: 675b8e0dcda0ebd0345d700f
paper_authors.name: Xin Wang
paper_authors._id: 675b8e0dcda0ebd0345d7010
paper_authors.name: Rachel Ward
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:38:50.889Z
paper_authors._id: 6626ab758a19c27042ed5e73
paper_authors.avatarUrl: /avatars/757f1fda0ce88a344d85bde05e468d9c.svg
paper_authors.fullname: Rachel Ward
paper_authors.type: user
paper_authors.user: rward314
paper_authors._id: 675b8e0dcda0ebd0345d7011
paper_authors.name: Yue Wu
paper_authors._id: 675b8e0dcda0ebd0345d7012
paper_authors.name: Dingli Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:39:05.875Z
paper_authors._id: 646bb050e96a751c5253955a
paper_authors.avatarUrl: /avatars/8bf9c373e1b2b990998eb4643a71549b.svg
paper_authors.fullname: Dingli Yu
paper_authors.type: user
paper_authors.user: dingliyu
paper_authors._id: 675b8e0dcda0ebd0345d7013
paper_authors.name: Cyril Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:39:13.873Z
paper_authors._id: 63a6a3a93d8cc9b1183c6593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a6a3a93d8cc9b1183c6593/UF-9ostqBHiAGDOuEtIw_.png
paper_authors.fullname: Cyril Zhang
paper_authors.type: user
paper_authors.user: cyrilzhang
paper_authors._id: 675b8e0dcda0ebd0345d7014
paper_authors.name: Yi Zhang
paper_title: Phi-4 Technical Report
paper_summary: We present phi-4, a 14-billion parameter language model developed with a
  training recipe that is centrally focused on data quality. Unlike most language
  models, where pre-training is based primarily on organic data sources such as
  web content or code, phi-4 strategically incorporates synthetic data throughout
  the training process. While previous models in the Phi family largely distill
  the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
  surpasses its teacher model on STEM-focused QA capabilities, giving evidence
  that our data-generation and post-training techniques go beyond distillation.
  Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
  performance relative to its size -- especially on reasoning-focused benchmarks
  -- due to improved data, training curriculum, and innovations in the
  post-training scheme.
paper_ai_keywords: language model 14-billion parameter training recipe data quality synthetic data pre-training organic data sources Phi family distill teacher model GPT-4 STEM-focused QA capabilities data-generation post-training techniques phi-3 architecture improved data training curriculum reasoning-focused benchmarks