title: Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language
    Models via Mixture-of-LoRAs
summary: We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable
  language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language
  model trained on high-quality web and synthetic data, significantly
  outperforming recent open-source models of similar size and matching the
  performance of models twice its size on math and coding tasks requiring complex
  reasoning. This achievement is driven by a carefully curated synthetic data
  recipe emphasizing high-quality math and coding datasets. Compared to its
  predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of
  200K tokens to better support multilingual applications, as well as group query
  attention for more efficient long-sequence generation. Phi-4-Multimodal is a
  multimodal model that integrates text, vision, and speech/audio input
  modalities into a single model. Its novel modality extension approach leverages
  LoRA adapters and modality-specific routers to allow multiple inference modes
  combining various modalities without interference. For example, it now ranks
  first in the OpenASR leaderboard to date, although the LoRA component of the
  speech/audio modality has just 460 million parameters. Phi-4-Multimodal
  supports scenarios involving (vision + language), (vision + speech), and
  (speech/audio) inputs, outperforming larger vision-language and speech-language
  models on a wide range of tasks. Additionally, we experiment to further train
  Phi-4-Mini to enhance its reasoning capabilities. Despite its compact
  3.8-billion-parameter size, this experimental version achieves reasoning
  performance on par with or surpassing significantly larger models, including
  DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
paper_authors._id: 67c67d0dfe135a5f482599bb
paper_authors.name: Abdelrahman Abouelenin
paper_authors._id: 67c67d0dfe135a5f482599bc
paper_authors.name: Atabak Ashfaq
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:45:15.511Z
paper_authors._id: 669ed17498ba26df962584f5
paper_authors.avatarUrl: /avatars/996c9cf05a4f8e5447552220085157c7.svg
paper_authors.fullname: Atabak Ashfaq
paper_authors.type: user
paper_authors.user: atabakashfaqMSFT
paper_authors._id: 67c67d0dfe135a5f482599bd
paper_authors.name: Adam Atkinson
paper_authors._id: 67c67d0dfe135a5f482599be
paper_authors.name: Hany Awadalla
paper_authors._id: 67c67d0dfe135a5f482599bf
paper_authors.name: Nguyen Bach
paper_authors._id: 67c67d0dfe135a5f482599c0
paper_authors.name: Jianmin Bao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:46:34.578Z
paper_authors._id: 6481e690f9ed842838a2b106
paper_authors.avatarUrl: /avatars/e89a3c8366df504a95dc08a1a412bf3d.svg
paper_authors.fullname: Jianmin Bao
paper_authors.type: user
paper_authors.user: jianmin-ustc
paper_authors._id: 67c67d0dfe135a5f482599c1
paper_authors.name: Alon Benhaim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:46:41.117Z
paper_authors._id: 65b9b627e7c838136275a681
paper_authors.avatarUrl: /avatars/22423f3d9a6c4ee34cad3b0894d27d23.svg
paper_authors.fullname: Alon Benhaim
paper_authors.type: user
paper_authors.user: alonbenhaim
paper_authors._id: 67c67d0dfe135a5f482599c2
paper_authors.name: Martin Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:46:47.556Z
paper_authors._id: 66f81b5b3c7ffa7931b4829a
paper_authors.avatarUrl: /avatars/a7f34e8e3fd92fdb96affc367b522fbe.svg
paper_authors.fullname: cai
paper_authors.type: user
paper_authors.user: martincai
paper_authors._id: 67c67d0dfe135a5f482599c3
paper_authors.name: Vishrav Chaudhary
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:46:56.428Z
paper_authors._id: 659c7ac977ac6f1bf5e63d7e
paper_authors.avatarUrl: /avatars/86a6efde0d483564a67ed5f344d479a0.svg
paper_authors.fullname: Vishrav Chaudhary
paper_authors.type: user
paper_authors.user: vishravmsft
paper_authors._id: 67c67d0dfe135a5f482599c4
paper_authors.name: Congcong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:04.205Z
paper_authors._id: 66c7a93b92e9f5b19f7533ab
paper_authors.avatarUrl: /avatars/e26ebf5cf083a3ec09fce24026ecc76e.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: congcongchen
paper_authors._id: 67c67d0dfe135a5f482599c5
paper_authors.name: Dong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:11.865Z
paper_authors._id: 666470a28f5513b0cf11e850
paper_authors.avatarUrl: /avatars/7beea758882677ad32a12ce56d4d084a.svg
paper_authors.fullname: Dong Chen
paper_authors.type: user
paper_authors.user: DongChen06
paper_authors._id: 67c67d0dfe135a5f482599c6
paper_authors.name: Dongdong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:18.197Z
paper_authors._id: 6567651c6fcc82e5e8c36d4d
paper_authors.avatarUrl: /avatars/ba3cc037a7688c4f8d967fc6043e540d.svg
paper_authors.fullname: Dongdong Chen
paper_authors.type: user
paper_authors.user: dongdongchen
paper_authors._id: 67c67d0dfe135a5f482599c7
paper_authors.name: Junkun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:43.236Z
paper_authors._id: 669db44d61278f96d8c608a4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/pNWygpo_bOcWlx51LeyeO.png
paper_authors.fullname: Junkun Chen
paper_authors.type: user
paper_authors.user: shtpgshus
paper_authors._id: 67c67d0dfe135a5f482599c8
paper_authors.name: Weizhu Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:51.832Z
paper_authors._id: 64da876370446182be5b608d
paper_authors.avatarUrl: /avatars/e412fdc71404ecdf638e416846e3ebfb.svg
paper_authors.fullname: Weizhu Chen
paper_authors.type: user
paper_authors.user: chenweizhu
paper_authors._id: 67c67d0dfe135a5f482599c9
paper_authors.name: Yen-Chun Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:47:58.051Z
paper_authors._id: 662d6b09a47b4da4b23c8b2a
paper_authors.avatarUrl: /avatars/6770b1d7e25b2cdce04f9904b543d122.svg
paper_authors.fullname: Yen-Chun Chen
paper_authors.type: user
paper_authors.user: Yen-ChunChen
paper_authors._id: 67c67d0dfe135a5f482599ca
paper_authors.name: Yi-ling Chen
paper_authors._id: 67c67d0dfe135a5f482599cb
paper_authors.name: Qi Dai
paper_authors._id: 67c67d0dfe135a5f482599cc
paper_authors.name: Xiyang Dai
paper_authors._id: 67c67d0dfe135a5f482599cd
paper_authors.name: Ruchao Fan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:40:17.936Z
paper_authors._id: 64a8b800b35f48e37dfd20fe
paper_authors.avatarUrl: /avatars/1e66be9a5238ce86df8b54150520bcc8.svg
paper_authors.fullname: Ruchao Fan
paper_authors.type: user
paper_authors.user: fanruchao
paper_authors._id: 67c67d0dfe135a5f482599ce
paper_authors.name: Mei Gao
paper_authors._id: 67c67d0dfe135a5f482599cf
paper_authors.name: Min Gao
paper_authors._id: 67c67d0dfe135a5f482599d0
paper_authors.name: Amit Garg
paper_authors._id: 67c67d0dfe135a5f482599d1
paper_authors.name: Abhishek Goswami
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:49:02.466Z
paper_authors._id: 62cdae333529c21a2283a0a1
paper_authors.avatarUrl: /avatars/cafc2821e522bbd06d49830e36a073e3.svg
paper_authors.fullname: Abhishek GOSWAMI
paper_authors.type: user
paper_authors.user: abgoswam
paper_authors._id: 67c67d0dfe135a5f482599d2
paper_authors.name: Junheng Hao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:53:16.356Z
paper_authors._id: 5f04c4394ec31d33a72116d6
paper_authors.avatarUrl: /avatars/75d4b9020070e73604b12e5adc1c8201.svg
paper_authors.fullname: Junheng Hao
paper_authors.type: user
paper_authors.user: jeffhao
paper_authors._id: 67c67d0dfe135a5f482599d3
paper_authors.name: Amr Hendy
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:53:24.716Z
paper_authors._id: 660480db07619487a3718a16
paper_authors.avatarUrl: /avatars/9c08d541913e57fd79988ef93d5095d4.svg
paper_authors.fullname: Amr Hendy
paper_authors.type: user
paper_authors.user: amrhendy
paper_authors._id: 67c67d0dfe135a5f482599d4
paper_authors.name: Yuxuan Hu
paper_authors._id: 67c67d0dfe135a5f482599d5
paper_authors.name: Xin Jin
paper_authors._id: 67c67d0dfe135a5f482599d6
paper_authors.name: Mahmoud Khademi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:53:53.225Z
paper_authors._id: 6440905e27dc46cca590994c
paper_authors.avatarUrl: /avatars/0346f8ad17038fba87649a0fc59d64ab.svg
paper_authors.fullname: Mahmoud Khademi
paper_authors.type: user
paper_authors.user: mkhademi
paper_authors._id: 67c67d0dfe135a5f482599d7
paper_authors.name: Dongwoo Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:54:04.257Z
paper_authors._id: 662476aec8920ec351b8d3d8
paper_authors.avatarUrl: /avatars/791e40f53073563680ef18f75b3ea95e.svg
paper_authors.fullname: Dongwoo Kim
paper_authors.type: user
paper_authors.user: dongwookim-ms
paper_authors._id: 67c67d0dfe135a5f482599d8
paper_authors.name: Young Jin Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:40:19.902Z
paper_authors._id: 63f5173bb51da4d61da6c038
paper_authors.avatarUrl: /avatars/0ee530cf80476aa3985c4d591cd384a1.svg
paper_authors.fullname: Young Jin Kim
paper_authors.type: user
paper_authors.user: ykim362
paper_authors._id: 67c67d0dfe135a5f482599d9
paper_authors.name: Gina Lee
paper_authors._id: 67c67d0dfe135a5f482599da
paper_authors.name: Jinyu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:54:17.115Z
paper_authors._id: 64004b72330a45b03604303b
paper_authors.avatarUrl: /avatars/a1fa3fc700173238d0336258b000d934.svg
paper_authors.fullname: Jinyu Li
paper_authors.type: user
paper_authors.user: FallTraveler
paper_authors._id: 67c67d0dfe135a5f482599db
paper_authors.name: Yunsheng Li
paper_authors._id: 67c67d0dfe135a5f482599dc
paper_authors.name: Chen Liang
paper_authors._id: 67c67d0dfe135a5f482599dd
paper_authors.name: Xihui Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:56:29.024Z
paper_authors._id: 6464f05e5cdb9ab50f846c98
paper_authors.avatarUrl: /avatars/3cb2f60a909b59289209ecc7ba75a338.svg
paper_authors.fullname: Xihui Lin
paper_authors.type: user
paper_authors.user: linxihui
paper_authors._id: 67c67d0dfe135a5f482599de
paper_authors.name: Zeqi Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:56:38.534Z
paper_authors._id: 62c3a0caf5e2eb44f51de87d
paper_authors.avatarUrl: /avatars/3c535c5488476b75443666176fcb4c9b.svg
paper_authors.fullname: Zeqi Lin
paper_authors.type: user
paper_authors.user: linzeqi
paper_authors._id: 67c67d0dfe135a5f482599df
paper_authors.name: Mengchen Liu
paper_authors._id: 67c67d0dfe135a5f482599e0
paper_authors.name: Yang Liu
paper_authors._id: 67c67d0dfe135a5f482599e1
paper_authors.name: Gilsinia Lopez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:59:55.169Z
paper_authors._id: 60c790f1accf7da31ed8240d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60c790f1accf7da31ed8240d/YDohCmgf9OUeWqZIs3Thh.jpeg
paper_authors.fullname: Gilsinia Lopez
paper_authors.type: user
paper_authors.user: lgg
paper_authors._id: 67c67d0dfe135a5f482599e2
paper_authors.name: Chong Luo
paper_authors._id: 67c67d0dfe135a5f482599e3
paper_authors.name: Piyush Madan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:02:38.019Z
paper_authors._id: 66269a329014ef4d10f55d9d
paper_authors.avatarUrl: /avatars/d4866c32419a7dd07e9aa0660f4bafa9.svg
paper_authors.fullname: Piyush Madan
paper_authors.type: user
paper_authors.user: PiyushMadan
paper_authors._id: 67c67d0dfe135a5f482599e4
paper_authors.name: Vadim Mazalov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:02:47.430Z
paper_authors._id: 65301591944086d1d5fcf656
paper_authors.avatarUrl: /avatars/250a2e898a4fcbe78feaf6e812851bd6.svg
paper_authors.fullname: Vadim Mazalovskii
paper_authors.type: user
paper_authors.user: JakeRiley
paper_authors._id: 67c67d0dfe135a5f482599e5
paper_authors.name: Ali Mousavi
paper_authors._id: 67c67d0dfe135a5f482599e6
paper_authors.name: Anh Nguyen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:57:52.311Z
paper_authors._id: 649bc84833486cdd77c01c66
paper_authors.avatarUrl: /avatars/36f4e4bb15c337c4391bfbd234051f4c.svg
paper_authors.fullname: Nguyen Anh
paper_authors.type: user
paper_authors.user: Anhnguyen
paper_authors._id: 67c67d0dfe135a5f482599e7
paper_authors.name: Jing Pan
paper_authors._id: 67c67d0dfe135a5f482599e8
paper_authors.name: Daniel Perez-Becker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:59:09.929Z
paper_authors._id: 673b7f70cdc852f69bebfed1
paper_authors.avatarUrl: /avatars/1efad61a42b948c750c96472a6192de5.svg
paper_authors.fullname: Daniel Perez-Becker
paper_authors.type: user
paper_authors.user: perezbecker
paper_authors._id: 67c67d0dfe135a5f482599e9
paper_authors.name: Jacob Platin
paper_authors._id: 67c67d0dfe135a5f482599ea
paper_authors.name: Thomas Portet
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:59:39.865Z
paper_authors._id: 65c52dad286bf45e79491697
paper_authors.avatarUrl: /avatars/01ebc7979273df6e53971ae9835b503f.svg
paper_authors.fullname: Thomas Portet
paper_authors.type: user
paper_authors.user: thopo
paper_authors._id: 67c67d0dfe135a5f482599eb
paper_authors.name: Kai Qiu
paper_authors._id: 67c67d0dfe135a5f482599ec
paper_authors.name: Bo Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:40:15.919Z
paper_authors._id: 668dcf92835bf7e64bbca904
paper_authors.avatarUrl: /avatars/416eb3a3c5318a6a45aad87012296470.svg
paper_authors.fullname: Bo Ren
paper_authors.type: user
paper_authors.user: rosrad
paper_authors._id: 67c67d0dfe135a5f482599ed
paper_authors.name: Liliang Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:57:37.996Z
paper_authors._id: 63815eff4761ddfa00903762
paper_authors.avatarUrl: /avatars/3419b239d42e091586f1c51b526d88e5.svg
paper_authors.fullname: Liliang Ren
paper_authors.type: user
paper_authors.user: renll
paper_authors._id: 67c67d0dfe135a5f482599ee
paper_authors.name: Sambuddha Roy
paper_authors._id: 67c67d0dfe135a5f482599ef
paper_authors.name: Ning Shang
paper_authors._id: 67c67d0dfe135a5f482599f0
paper_authors.name: Yelong Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T10:00:05.457Z
paper_authors._id: 6454c337a13edf669cd5d8ea
paper_authors.avatarUrl: /avatars/a383a0dda7c2ef6a0d6c3c64651f42ff.svg
paper_authors.fullname: Yelong Shen
paper_authors.type: user
paper_authors.user: uuu6
paper_authors._id: 67c67d0dfe135a5f482599f1
paper_authors.name: Saksham Singhal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:59:03.188Z
paper_authors._id: 62743aec8cb70eed79073bc0
paper_authors.avatarUrl: /avatars/3c8b9a91d898f616265f823ab7d432df.svg
paper_authors.fullname: Saksham Singhal
paper_authors.type: user
paper_authors.user: sakshamsinghal
paper_authors._id: 67c67d0dfe135a5f482599f2
paper_authors.name: Subhojit Som
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:59:47.241Z
paper_authors._id: 678bc6b432ee4968eca9bb6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/wT-Xa3TYem_EzkZZMyDG0.png
paper_authors.fullname: Subhojit Som
paper_authors.type: user
paper_authors.user: susom
paper_authors._id: 67c67d0dfe135a5f482599f3
paper_authors.name: Xia Song
paper_authors._id: 67c67d0dfe135a5f482599f4
paper_authors.name: Tetyana Sych
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:58:27.814Z
paper_authors._id: 64692ad25d701566394fd8da
paper_authors.avatarUrl: /avatars/d6811ccceb14788bfa0aa10fe4ee1054.svg
paper_authors.fullname: Tetyana Sych
paper_authors.type: user
paper_authors.user: tesych
paper_authors._id: 67c67d0dfe135a5f482599f5
paper_authors.name: Praneetha Vaddamanu
paper_authors._id: 67c67d0dfe135a5f482599f6
paper_authors.name: Shuohang Wang
paper_authors._id: 67c67d0dfe135a5f482599f7
paper_authors.name: Yiming Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T21:16:18.278Z
paper_authors._id: 6786f93b3ad5585f2c2828b1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6786f93b3ad5585f2c2828b1/xbAqCmsuAcwClUj_lAPHi.jpeg
paper_authors.fullname: Yiming Wang
paper_authors.type: user
paper_authors.user: freewym
paper_authors._id: 67c67d0dfe135a5f482599f8
paper_authors.name: Zhenghao Wang
paper_authors._id: 67c67d0dfe135a5f482599f9
paper_authors.name: Haibin Wu
paper_authors._id: 67c67d0dfe135a5f482599fa
paper_authors.name: Haoran Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:56:04.939Z
paper_authors._id: 61384b860317b0a5c10877d3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631080954171-61384b860317b0a5c10877d3.jpeg
paper_authors.fullname: Haoran Xu
paper_authors.type: user
paper_authors.user: haoranxu
paper_authors._id: 67c67d0dfe135a5f482599fb
paper_authors.name: Weijian Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:58:36.082Z
paper_authors._id: 6398f4b32c20654083f36cde
paper_authors.avatarUrl: /avatars/4591f514483890997c55e9e6d60bbb0f.svg
paper_authors.fullname: Weijian Xu
paper_authors.type: user
paper_authors.user: xwjabc
paper_authors._id: 67c67d0dfe135a5f482599fc
paper_authors.name: Yifan Yang
paper_authors._id: 67c67d0dfe135a5f482599fd
paper_authors.name: Ziyi Yang
paper_authors._id: 67c67d0dfe135a5f482599fe
paper_authors.name: Donghan Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:55:41.798Z
paper_authors._id: 65b01b8a29ae836e9ed5af24
paper_authors.avatarUrl: /avatars/a8b78a4b54d3f10858c5925521357001.svg
paper_authors.fullname: Donghan Yu
paper_authors.type: user
paper_authors.user: donghanyu
paper_authors._id: 67c67d0dfe135a5f482599ff
paper_authors.name: Ishmam Zabir
paper_authors._id: 67c67d0dfe135a5f48259a00
paper_authors.name: Jianwen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:55:12.465Z
paper_authors._id: 63601ee38fb9c2420ffbe45d
paper_authors.avatarUrl: /avatars/56af091aaff1b42dcfbae84a6ee1e7f7.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Jianwen
paper_authors._id: 67c67d0dfe135a5f48259a01
paper_authors.name: Li Lyna Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:55:01.540Z
paper_authors._id: 62b0009c72043b05d29492b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b0009c72043b05d29492b2/NqRkX2YLhlfOLvYysa7dD.png
paper_authors.fullname: Li Lyna Zhang
paper_authors.type: user
paper_authors.user: lynazhang
paper_authors._id: 67c67d0dfe135a5f48259a02
paper_authors.name: Yunan Zhang
paper_authors._id: 67c67d0dfe135a5f48259a03
paper_authors.name: Xiren Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-04T09:54:26.629Z
paper_authors._id: 66ce4c9f864befb39cfc74e9
paper_authors.avatarUrl: /avatars/ef66398466c470fc1d384c6817d9e461.svg
paper_authors.fullname: Xiren Zhou
paper_authors.type: user
paper_authors.user: XirenZhou
paper_title: Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language
    Models via Mixture-of-LoRAs
paper_summary: We introduce Phi-4-Mini and Phi-4-Multimodal, compact yet highly capable
  language and multimodal models. Phi-4-Mini is a 3.8-billion-parameter language
  model trained on high-quality web and synthetic data, significantly
  outperforming recent open-source models of similar size and matching the
  performance of models twice its size on math and coding tasks requiring complex
  reasoning. This achievement is driven by a carefully curated synthetic data
  recipe emphasizing high-quality math and coding datasets. Compared to its
  predecessor, Phi-3.5-Mini, Phi-4-Mini features an expanded vocabulary size of
  200K tokens to better support multilingual applications, as well as group query
  attention for more efficient long-sequence generation. Phi-4-Multimodal is a
  multimodal model that integrates text, vision, and speech/audio input
  modalities into a single model. Its novel modality extension approach leverages
  LoRA adapters and modality-specific routers to allow multiple inference modes
  combining various modalities without interference. For example, it now ranks
  first in the OpenASR leaderboard to date, although the LoRA component of the
  speech/audio modality has just 460 million parameters. Phi-4-Multimodal
  supports scenarios involving (vision + language), (vision + speech), and
  (speech/audio) inputs, outperforming larger vision-language and speech-language
  models on a wide range of tasks. Additionally, we experiment to further train
  Phi-4-Mini to enhance its reasoning capabilities. Despite its compact
  3.8-billion-parameter size, this experimental version achieves reasoning
  performance on par with or surpassing significantly larger models, including
  DeepSeek-R1-Distill-Qwen-7B and DeepSeek-R1-Distill-Llama-8B.
paper_ai_keywords: group query attention LoRA adapters modality-specific routers OpenASR leaderboard vision-language speech-language DeepSeek-R1-Distill-Qwen-7B DeepSeek-R1-Distill-Llama-8B