Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language
  Models via Mixture-of-LoRAs
We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable
language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language
model trained on high-quality web and synthetic data, significantly
outperforming recent open-source models of similar size and matching the
performance of models twice its size on math and coding tasks requiring complex
reasoning. This achievement is driven by a carefully curated synthetic data
recipe emphasizing high-quality math and coding datasets. Compared to its
predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of
200K tokens to better support multilingual applications, as well as group query
attention for more efficient long-sequence generation. Phi-4-Multimodal is a
multimodal model that integrates text, vision, and speech/audio input
modalities into a single model. Its novel modality extension approach leverages
LoRA adapters and modality-specific routers to allow multiple inference modes
combining various modalities without interference. For example, it now ranks
first in the OpenASR leaderboard to date, although the LoRA component of the
speech/audio modality has just 460 million parameters. Phi-4-Multimodal
supports scenarios involving (vision + language), (vision + speech), and
(speech/audio) inputs, outperforming larger vision-language and speech-language
models on a wide range of tasks. Additionally, we experiment to further train
Phi-4-Mini to enhance its reasoning capabilities. Despite its compact
3.8-billion-parameter size, this experimental version achieves reasoning
performance on par with or surpassing significantly larger models, including
DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
67c67d0dfe135a5f482599bb
Abdelrahman Abouelenin
67c67d0dfe135a5f482599bc
Atabak Ashfaq
admin_assigned
2025-03-04T09:45:15.511Z
669ed17498ba26df962584f5
/avatars/996c9cf05a4f8e5447552220085157c7.svg
Atabak Ashfaq
user
atabakashfaqMSFT
67c67d0dfe135a5f482599bd
Adam Atkinson
67c67d0dfe135a5f482599be
Hany Awadalla
67c67d0dfe135a5f482599bf
Nguyen Bach
67c67d0dfe135a5f482599c0
Jianmin Bao
admin_assigned
2025-03-04T09:46:34.578Z
6481e690f9ed842838a2b106
/avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
Jianmin Bao
user
jianmin-ustc
67c67d0dfe135a5f482599c1
Alon Benhaim
admin_assigned
2025-03-04T09:46:41.117Z
65b9b627e7c838136275a681
/avatars/22423f3d9a6c4ee34cad3b0894d27d23.svg
Alon Benhaim
user
alonbenhaim
67c67d0dfe135a5f482599c2
Martin Cai
admin_assigned
2025-03-04T09:46:47.556Z
66f81b5b3c7ffa7931b4829a
/avatars/a7f34e8e3fd92fdb96affc367b522fbe.svg
cai
user
martincai
67c67d0dfe135a5f482599c3
Vishrav Chaudhary
admin_assigned
2025-03-04T09:46:56.428Z
659c7ac977ac6f1bf5e63d7e
/avatars/86a6efde0d483564a67ed5f344d479a0.svg
Vishrav Chaudhary
user
vishravmsft
67c67d0dfe135a5f482599c4
Congcong Chen
admin_assigned
2025-03-04T09:47:04.205Z
66c7a93b92e9f5b19f7533ab
/avatars/e26ebf5cf083a3ec09fce24026ecc76e.svg
Chen
user
congcongchen
67c67d0dfe135a5f482599c5
Dong Chen
admin_assigned
2025-03-04T09:47:11.865Z
666470a28f5513b0cf11e850
/avatars/7beea758882677ad32a12ce56d4d084a.svg
Dong Chen
user
DongChen06
67c67d0dfe135a5f482599c6
Dongdong Chen
admin_assigned
2025-03-04T09:47:18.197Z
6567651c6fcc82e5e8c36d4d
/avatars/ba3cc037a7688c4f8d967fc6043e540d.svg
Dongdong Chen
user
dongdongchen
67c67d0dfe135a5f482599c7
Junkun Chen
admin_assigned
2025-03-04T09:47:43.236Z
669db44d61278f96d8c608a4
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pNWygpo_bOcWlx51LeyeO.png
Junkun Chen
user
shtpgshus
67c67d0dfe135a5f482599c8
Weizhu Chen
admin_assigned
2025-03-04T09:47:51.832Z
64da876370446182be5b608d
/avatars/e412fdc71404ecdf638e416846e3ebfb.svg
Weizhu Chen
user
chenweizhu
67c67d0dfe135a5f482599c9
Yen-Chun Chen
admin_assigned
2025-03-04T09:47:58.051Z
662d6b09a47b4da4b23c8b2a
/avatars/6770b1d7e25b2cdce04f9904b543d122.svg
Yen-Chun Chen
user
Yen-ChunChen
67c67d0dfe135a5f482599ca
Yi-ling Chen
67c67d0dfe135a5f482599cb
Qi Dai
67c67d0dfe135a5f482599cc
Xiyang Dai
67c67d0dfe135a5f482599cd
Ruchao Fan
claimed_verified
2025-03-04T08:40:17.936Z
64a8b800b35f48e37dfd20fe
/avatars/1e66be9a5238ce86df8b54150520bcc8.svg
Ruchao Fan
user
fanruchao
67c67d0dfe135a5f482599ce
Mei Gao
67c67d0dfe135a5f482599cf
Min Gao
67c67d0dfe135a5f482599d0
Amit Garg
67c67d0dfe135a5f482599d1
Abhishek Goswami
admin_assigned
2025-03-04T09:49:02.466Z
62cdae333529c21a2283a0a1
/avatars/cafc2821e522bbd06d49830e36a073e3.svg
Abhishek GOSWAMI
user
abgoswam
67c67d0dfe135a5f482599d2
Junheng Hao
admin_assigned
2025-03-04T09:53:16.356Z
5f04c4394ec31d33a72116d6
/avatars/75d4b9020070e73604b12e5adc1c8201.svg
Junheng Hao
user
jeffhao
67c67d0dfe135a5f482599d3
Amr Hendy
admin_assigned
2025-03-04T09:53:24.716Z
660480db07619487a3718a16
/avatars/9c08d541913e57fd79988ef93d5095d4.svg
Amr Hendy
user
amrhendy
67c67d0dfe135a5f482599d4
Yuxuan Hu
67c67d0dfe135a5f482599d5
Xin Jin
67c67d0dfe135a5f482599d6
Mahmoud Khademi
admin_assigned
2025-03-04T09:53:53.225Z
6440905e27dc46cca590994c
/avatars/0346f8ad17038fba87649a0fc59d64ab.svg
Mahmoud Khademi
user
mkhademi
67c67d0dfe135a5f482599d7
Dongwoo Kim
admin_assigned
2025-03-04T09:54:04.257Z
662476aec8920ec351b8d3d8
/avatars/791e40f53073563680ef18f75b3ea95e.svg
Dongwoo Kim
user
dongwookim-ms
67c67d0dfe135a5f482599d8
Young Jin Kim
claimed_verified
2025-03-04T08:40:19.902Z
63f5173bb51da4d61da6c038
/avatars/0ee530cf80476aa3985c4d591cd384a1.svg
Young Jin Kim
user
ykim362
67c67d0dfe135a5f482599d9
Gina Lee
claimed_verified
2025-07-22T07:54:58.969Z
66c3a59db01b19d8c31e4fe8
/avatars/b36d84168d7e17ba6cf8b5b3d4a774e7.svg
Gina Lee
user
gl0819
67c67d0dfe135a5f482599da
Jinyu Li
admin_assigned
2025-03-04T09:54:17.115Z
64004b72330a45b03604303b
/avatars/a1fa3fc700173238d0336258b000d934.svg
Jinyu Li
user
FallTraveler
67c67d0dfe135a5f482599db
Yunsheng Li
67c67d0dfe135a5f482599dc
Chen Liang
67c67d0dfe135a5f482599dd
Xihui Lin
admin_assigned
2025-03-04T09:56:29.024Z
6464f05e5cdb9ab50f846c98
/avatars/3cb2f60a909b59289209ecc7ba75a338.svg
Xihui Lin
user
linxihui
67c67d0dfe135a5f482599de
Zeqi Lin
admin_assigned
2025-03-04T09:56:38.534Z
62c3a0caf5e2eb44f51de87d
/avatars/3c535c5488476b75443666176fcb4c9b.svg
Zeqi Lin
user
linzeqi
67c67d0dfe135a5f482599df
Mengchen Liu
67c67d0dfe135a5f482599e0
Yang Liu
67c67d0dfe135a5f482599e1
Gilsinia Lopez
admin_assigned
2025-03-04T09:59:55.169Z
60c790f1accf7da31ed8240d
https://cdn-avatars.huggingface.co/v1/production/uploads/60c790f1accf7da31ed8240d/YDohCmgf9OUeWqZIs3Thh.jpeg
Gilsinia Lopez
user
lgg
67c67d0dfe135a5f482599e2
Chong Luo
67c67d0dfe135a5f482599e3
Piyush Madan
admin_assigned
2025-03-04T10:02:38.019Z
66269a329014ef4d10f55d9d
/avatars/d4866c32419a7dd07e9aa0660f4bafa9.svg
Piyush Madan
user
PiyushMadan
67c67d0dfe135a5f482599e4
Vadim Mazalov
admin_assigned
2025-03-04T10:02:47.430Z
65301591944086d1d5fcf656
/avatars/250a2e898a4fcbe78feaf6e812851bd6.svg
Vadim Mazalovskii
user
JakeRiley
67c67d0dfe135a5f482599e5
Ali Mousavi
67c67d0dfe135a5f482599e6
Anh Nguyen
admin_assigned
2025-03-04T09:57:52.311Z
649bc84833486cdd77c01c66
/avatars/36f4e4bb15c337c4391bfbd234051f4c.svg
Nguyen Anh
user
Anhnguyen
67c67d0dfe135a5f482599e7
Jing Pan
67c67d0dfe135a5f482599e8
Daniel Perez-Becker
admin_assigned
2025-03-04T09:59:09.929Z
673b7f70cdc852f69bebfed1
/avatars/1efad61a42b948c750c96472a6192de5.svg
Daniel Perez-Becker
user
perezbecker
67c67d0dfe135a5f482599e9
Jacob Platin
67c67d0dfe135a5f482599ea
Thomas Portet
admin_assigned
2025-03-04T09:59:39.865Z
65c52dad286bf45e79491697
/avatars/01ebc7979273df6e53971ae9835b503f.svg
Thomas Portet
user
thopo
67c67d0dfe135a5f482599eb
Kai Qiu
67c67d0dfe135a5f482599ec
Bo Ren
claimed_verified
2025-03-04T08:40:15.919Z
668dcf92835bf7e64bbca904
/avatars/416eb3a3c5318a6a45aad87012296470.svg
Bo Ren
user
rosrad
67c67d0dfe135a5f482599ed
Liliang Ren
admin_assigned
2025-03-04T09:57:37.996Z
63815eff4761ddfa00903762
https://cdn-avatars.huggingface.co/v1/production/uploads/63815eff4761ddfa00903762/YlkfX3sTZRQcoapctXJeJ.jpeg
Liliang Ren
user
renll
67c67d0dfe135a5f482599ee
Sambuddha Roy
67c67d0dfe135a5f482599ef
Ning Shang
67c67d0dfe135a5f482599f0
Yelong Shen
admin_assigned
2025-03-04T10:00:05.457Z
6454c337a13edf669cd5d8ea
/avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
Yelong Shen
user
uuu6
67c67d0dfe135a5f482599f1
Saksham Singhal
admin_assigned
2025-03-04T09:59:03.188Z
62743aec8cb70eed79073bc0
/avatars/3c8b9a91d898f616265f823ab7d432df.svg
Saksham Singhal
user
sakshamsinghal
67c67d0dfe135a5f482599f2
Subhojit Som
admin_assigned
2025-03-04T09:59:47.241Z
678bc6b432ee4968eca9bb6a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wT-Xa3TYem_EzkZZMyDG0.png
Subhojit Som
user
susom
67c67d0dfe135a5f482599f3
Xia Song
67c67d0dfe135a5f482599f4
Tetyana Sych
admin_assigned
2025-03-04T09:58:27.814Z
64692ad25d701566394fd8da
/avatars/d6811ccceb14788bfa0aa10fe4ee1054.svg
Tetyana Sych
user
tesych
67c67d0dfe135a5f482599f5
Praneetha Vaddamanu
67c67d0dfe135a5f482599f6
Shuohang Wang
67c67d0dfe135a5f482599f7
Yiming Wang
claimed_verified
2025-03-04T21:16:18.278Z
6786f93b3ad5585f2c2828b1
https://cdn-avatars.huggingface.co/v1/production/uploads/6786f93b3ad5585f2c2828b1/xbAqCmsuAcwClUj_lAPHi.jpeg
Yiming Wang
user
freewym
67c67d0dfe135a5f482599f8
Zhenghao Wang
67c67d0dfe135a5f482599f9
Haibin Wu
67c67d0dfe135a5f482599fa
Haoran Xu
admin_assigned
2025-03-04T09:56:04.939Z
61384b860317b0a5c10877d3
https://cdn-avatars.huggingface.co/v1/production/uploads/1631080954171-61384b860317b0a5c10877d3.jpeg
Haoran Xu
user
haoranxu
67c67d0dfe135a5f482599fb
Weijian Xu
admin_assigned
2025-03-04T09:58:36.082Z
6398f4b32c20654083f36cde
/avatars/4591f514483890997c55e9e6d60bbb0f.svg
Weijian Xu
user
xwjabc
67c67d0dfe135a5f482599fc
Yifan Yang
67c67d0dfe135a5f482599fd
Ziyi Yang
67c67d0dfe135a5f482599fe
Donghan Yu
admin_assigned
2025-03-04T09:55:41.798Z
65b01b8a29ae836e9ed5af24
/avatars/a8b78a4b54d3f10858c5925521357001.svg
Donghan Yu
user
donghanyu
67c67d0dfe135a5f482599ff
Ishmam Zabir
67c67d0dfe135a5f48259a00
Jianwen Zhang
admin_assigned
2025-03-04T09:55:12.465Z
63601ee38fb9c2420ffbe45d
/avatars/56af091aaff1b42dcfbae84a6ee1e7f7.svg
Jianwen Zhang
user
Jianwen
67c67d0dfe135a5f48259a01
Li Lyna Zhang
admin_assigned
2025-03-04T09:55:01.540Z
62b0009c72043b05d29492b2
https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
Li Lyna Zhang
user
lynazhang
67c67d0dfe135a5f48259a02
Yunan Zhang
67c67d0dfe135a5f48259a03
Xiren Zhou
admin_assigned
2025-03-04T09:54:26.629Z
66ce4c9f864befb39cfc74e9
/avatars/ef66398466c470fc1d384c6817d9e461.svg
Xiren Zhou
user
XirenZhou
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language
  Models via Mixture-of-LoRAs
We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable
language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language
model trained on high-quality web and synthetic data, significantly
outperforming recent open-source models of similar size and matching the
performance of models twice its size on math and coding tasks requiring complex
reasoning. This achievement is driven by a carefully curated synthetic data
recipe emphasizing high-quality math and coding datasets. Compared to its
predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of
200K tokens to better support multilingual applications, as well as group query
attention for more efficient long-sequence generation. Phi-4-Multimodal is a
multimodal model that integrates text, vision, and speech/audio input
modalities into a single model. Its novel modality extension approach leverages
LoRA adapters and modality-specific routers to allow multiple inference modes
combining various modalities without interference. For example, it now ranks
first in the OpenASR leaderboard to date, although the LoRA component of the
speech/audio modality has just 460 million parameters. Phi-4-Multimodal
supports scenarios involving (vision + language), (vision + speech), and
(speech/audio) inputs, outperforming larger vision-language and speech-language
models on a wide range of tasks. Additionally, we experiment to further train
Phi-4-Mini to enhance its reasoning capabilities. Despite its compact
3.8-billion-parameter size, this experimental version achieves reasoning
performance on par with or surpassing significantly larger models, including
DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
Phi-4-Mini and Phi-4-Multimodal are compact yet highly capable language and multimodal models, featuring advanced attention mechanisms and LoRA adapters for efficient and powerful reasoning across various tasks.
group query attention LoRA adapters modality-specific routers OpenASR DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Llama-8B