Phi-4 Technical Report
We present phi-4, a 14-billion parameter language model developed with a
training recipe that is centrally focused on data quality. Unlike most language
models, where pre-training is based primarily on organic data sources such as
web content or code, phi-4 strategically incorporates synthetic data throughout
the training process. While previous models in the Phi family largely distill
the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
surpasses its teacher model on STEM-focused QA capabilities, giving evidence
that our data-generation and post-training techniques go beyond distillation.
Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
performance relative to its size -- especially on reasoning-focused benchmarks
-- due to improved data, training curriculum, and innovations in the
post-training scheme.
675b8e0dcda0ebd0345d6ffa
Marah Abdin
admin_assigned
2024-12-16T10:34:10.073Z
649af1e3ca20306aeef590b8
https://cdn-avatars.huggingface.co/v1/production/uploads/649af1e3ca20306aeef590b8/Tvy8yf7zDO2FoNs03tWj6.jpeg
Marah Abdin
user
marah-abdin
675b8e0dcda0ebd0345d6ffb
Jyoti Aneja
admin_assigned
2024-12-16T10:34:16.014Z
623e17038972a8c030af23fa
https://cdn-avatars.huggingface.co/v1/production/uploads/623e17038972a8c030af23fa/2ERyhCNAdV2wt4uRy3Wq5.jpeg
Jyoti Aneja
user
jyotiA
675b8e0dcda0ebd0345d6ffc
Harkirat Behl
admin_assigned
2024-12-16T10:34:21.406Z
63c6aa9e656e7822e2359d9c
https://cdn-avatars.huggingface.co/v1/production/uploads/63c6aa9e656e7822e2359d9c/4rn_faJ23m_a6bevXEDG_.jpeg
Harkirat Behl
user
Harkirat
675b8e0dcda0ebd0345d6ffd
Sébastien Bubeck
admin_assigned
2024-12-16T10:34:27.076Z
64557408c9c0dcc8c24a7a92
/avatars/911ba8fc3c224578d5946dddd94bf4c0.svg
Sebastien Bubeck
user
sebubeck
675b8e0dcda0ebd0345d6ffe
Ronen Eldan
admin_assigned
2024-12-16T10:34:32.923Z
633723a80267ebcf0264c06b
/avatars/22bb971597e9f3abfa343280a9d0f65f.svg
Ronen Eldan
user
roneneldan
675b8e0dcda0ebd0345d6fff
Suriya Gunasekar
admin_assigned
2024-12-16T10:34:38.249Z
63e1b4f77fbb6ae4d4f36aa4
/avatars/7febc332d393e0bb94255104527ffe2a.svg
Gunasekar
user
suriyagunasekar
675b8e0dcda0ebd0345d7000
Michael Harrison
675b8e0dcda0ebd0345d7001
Russell J. Hewett
675b8e0dcda0ebd0345d7002
Mojan Javaheripi
admin_assigned
2024-12-16T10:35:16.168Z
643dbfa31e5be78c66440fcf
https://cdn-avatars.huggingface.co/v1/production/uploads/643dbfa31e5be78c66440fcf/VtLrAlPY_4ncVrM4eL7V-.jpeg
Mojan Javaheripi
user
mojanjp
675b8e0dcda0ebd0345d7003
Piero Kauffmann
admin_assigned
2024-12-16T10:35:24.374Z
662699f051cedbbb0b1a9e81
/avatars/8d29d2a7b9c7d714617e3961f204bfc6.svg
Piero Kauffmann
user
piero2c
675b8e0dcda0ebd0345d7004
James R. Lee
675b8e0dcda0ebd0345d7005
Yin Tat Lee
admin_assigned
2024-12-16T10:35:58.047Z
6303297604c75db08b8972e2
/avatars/693b2dada2244ba5d97df9518f473ccb.svg
Yin Tat Lee
user
yintat
675b8e0dcda0ebd0345d7006
Yuanzhi Li
admin_assigned
2024-12-16T10:36:18.888Z
641d5a313f04f9bf2da17e8c
/avatars/4035f73c69ddb045573ad96db4f13f04.svg
Yuanzhi Li
user
Uushizhu1234
675b8e0dcda0ebd0345d7007
Weishung Liu
admin_assigned
2024-12-16T10:36:27.196Z
637c3ac41c13d2970bf67625
https://cdn-avatars.huggingface.co/v1/production/uploads/637c3ac41c13d2970bf67625/K_KEPoWjj4k7EndHeU2Jt.png
Weishung Liu 
user
weish
675b8e0dcda0ebd0345d7008
Caio C. T. Mendes
675b8e0dcda0ebd0345d7009
Anh Nguyen
admin_assigned
2024-12-16T10:36:50.168Z
649bc84833486cdd77c01c66
/avatars/36f4e4bb15c337c4391bfbd234051f4c.svg
Nguyen Anh
user
Anhnguyen
675b8e0dcda0ebd0345d700a
Eric Price
675b8e0dcda0ebd0345d700b
Gustavo de Rosa
claimed_verified
2024-12-13T16:05:51.994Z
6157454831624da88210e627
https://cdn-avatars.huggingface.co/v1/production/uploads/1666203761402-6157454831624da88210e627.jpeg
Gustavo de Rosa
user
gugarosa
675b8e0dcda0ebd0345d700c
Olli Saarikivi
admin_assigned
2024-12-16T10:37:26.568Z
6477a7108ab7e732b6d86701
/avatars/3319db398ce8142bd7d61de1d2cd1b4b.svg
Olli Saarikivi
user
olsaarik
675b8e0dcda0ebd0345d700d
Adil Salim
admin_assigned
2024-12-16T10:37:31.780Z
670851dc39406c0d7e47c373
/avatars/f4bff694d2ce4665ad281ae5f5c3346a.svg
Adil Salim
user
AdilSlm
675b8e0dcda0ebd0345d700e
Shital Shah
admin_assigned
2024-12-16T10:38:03.800Z
5fea740dbb3114fa14c82934
/avatars/5f9fe9aaf74a3b930b6fcebcf8a4e4cc.svg
Shital Shah
user
sytelus
675b8e0dcda0ebd0345d700f
Xin Wang
675b8e0dcda0ebd0345d7010
Rachel Ward
admin_assigned
2024-12-16T10:38:50.889Z
6626ab758a19c27042ed5e73
/avatars/757f1fda0ce88a344d85bde05e468d9c.svg
Rachel Ward
user
rward314
675b8e0dcda0ebd0345d7011
Yue Wu
675b8e0dcda0ebd0345d7012
Dingli Yu
admin_assigned
2024-12-16T10:39:05.875Z
646bb050e96a751c5253955a
/avatars/8bf9c373e1b2b990998eb4643a71549b.svg
Dingli Yu
user
dingliyu
675b8e0dcda0ebd0345d7013
Cyril Zhang
admin_assigned
2024-12-16T10:39:13.873Z
63a6a3a93d8cc9b1183c6593
https://cdn-avatars.huggingface.co/v1/production/uploads/63a6a3a93d8cc9b1183c6593/UF-9ostqBHiAGDOuEtIw_.png
Cyril Zhang
user
cyrilzhang
675b8e0dcda0ebd0345d7014
Yi Zhang
Phi-4 Technical Report
We present phi-4, a 14-billion parameter language model developed with a
training recipe that is centrally focused on data quality. Unlike most language
models, where pre-training is based primarily on organic data sources such as
web content or code, phi-4 strategically incorporates synthetic data throughout
the training process. While previous models in the Phi family largely distill
the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
surpasses its teacher model on STEM-focused QA capabilities, giving evidence
that our data-generation and post-training techniques go beyond distillation.
Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
performance relative to its size -- especially on reasoning-focused benchmarks
-- due to improved data, training curriculum, and innovations in the
post-training scheme.
A 14-billion parameter language model surpasses its teacher model in STEM-focused QA through strategic use of synthetic data, improved data quality, and enhanced training techniques.
training recipe data quality synthetic data teacher model GPT-4 STEM-focused QA post-training techniques training curriculum