title: Configurable Foundation Models: Building LLMs from a Modular Perspective
summary: Advancements in LLMs have recently unveiled challenges tied to computational
  efficiency and continual scalability due to their requirements of huge
  parameters, making the applications and evolution of these models on devices
  with limited computation resources and scenarios requiring various abilities
  increasingly cumbersome. Inspired by modularity within the human brain, there
  is a growing tendency to decompose LLMs into numerous functional modules,
  allowing for inference with part of modules and dynamic assembly of modules to
  tackle complex tasks, such as mixture-of-experts. To highlight the inherent
  efficiency and composability of the modular approach, we coin the term brick to
  represent each functional module, designating the modularized structure as
  configurable foundation models. In this paper, we offer a comprehensive
  overview and investigation of the construction, utilization, and limitation of
  configurable foundation models. We first formalize modules into emergent bricks
  - functional neuron partitions that emerge during the pre-training phase, and
  customized bricks - bricks constructed via additional post-training to improve
  the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
  further present four brick-oriented operations: retrieval and routing, merging,
  updating, and growing. These operations allow for dynamic configuration of LLMs
  based on instructions to handle complex tasks. To verify our perspective, we
  conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
  follow modular patterns with functional specialization of neurons and
  functional neuron partitions. Finally, we highlight several open issues and
  directions for future research. Overall, this paper aims to offer a fresh
  modular perspective on existing LLM research and inspire the future creation of
  more efficient and scalable foundational models.
paper_authors._id: 66db3dfca46deac8798e3c95
paper_authors.name: Chaojun Xiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T06:55:38.495Z
paper_authors._id: 608f6d72283d0a8d7be9d1f9
paper_authors.avatarUrl: /avatars/7f499a37019359a3c488ba6cc11751fc.svg
paper_authors.fullname: Chaojun XIAO
paper_authors.type: user
paper_authors.user: xcjthu
paper_authors._id: 66db3dfca46deac8798e3c96
paper_authors.name: Zhengyan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:52:09.262Z
paper_authors._id: 65654ed2219af7f841640f27
paper_authors.avatarUrl: /avatars/e6904b3479fc5e65ea1f752919ca8290.svg
paper_authors.fullname: Zhengyan Zhang
paper_authors.type: user
paper_authors.user: ZhengyanZhang
paper_authors._id: 66db3dfca46deac8798e3c97
paper_authors.name: Chenyang Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:52:39.751Z
paper_authors._id: 64c09684e56520a63d35ec87
paper_authors.avatarUrl: /avatars/86a338e8d122a66a94143bbb9bf3ebf8.svg
paper_authors.fullname: Chenyang Song
paper_authors.type: user
paper_authors.user: Raincleared
paper_authors._id: 66db3dfca46deac8798e3c98
paper_authors.name: Dazhi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:54:28.930Z
paper_authors._id: 65ba471ad88a65abb9328ee2
paper_authors.avatarUrl: /avatars/956238ce5034091e64d026b0272c4400.svg
paper_authors.fullname: Dazhi Jiang
paper_authors.type: user
paper_authors.user: thuzhizhi
paper_authors._id: 66db3dfca46deac8798e3c99
paper_authors.name: Feng Yao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T06:55:40.389Z
paper_authors._id: 64c8b2c5c547ed5243e14a6e
paper_authors.avatarUrl: /avatars/96d4a9010f96001c8cff235915926390.svg
paper_authors.fullname: Feng Yao
paper_authors.type: user
paper_authors.user: fengyao1909
paper_authors._id: 66db3dfca46deac8798e3c9a
paper_authors.name: Xu Han
paper_authors._id: 66db3dfca46deac8798e3c9b
paper_authors.name: Xiaozhi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:54:55.877Z
paper_authors._id: 648c48e7a47a0850f9458b67
paper_authors.avatarUrl: /avatars/47d71d80f9901313feb0199c37296389.svg
paper_authors.fullname: Xiaozhi Wang
paper_authors.type: user
paper_authors.user: wangxz098
paper_authors._id: 66db3dfca46deac8798e3c9c
paper_authors.name: Shuo Wang
paper_authors._id: 66db3dfca46deac8798e3c9d
paper_authors.name: Yufei Huang
paper_authors._id: 66db3dfca46deac8798e3c9e
paper_authors.name: Guanyu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:56:03.905Z
paper_authors._id: 6586a7a84f9d2b955ee21b0d
paper_authors.avatarUrl: /avatars/4f55c26fe62a12aab8f1413387d2e75a.svg
paper_authors.fullname: Guanyu Lin
paper_authors.type: user
paper_authors.user: cmulgy
paper_authors._id: 66db3dfca46deac8798e3c9f
paper_authors.name: Yingfa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:56:20.870Z
paper_authors._id: 6144e4667f2544bb450787b2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6144e4667f2544bb450787b2/5wSDDqJbI4TGtBLP9IvjY.png
paper_authors.fullname: Yingfa Chen
paper_authors.type: user
paper_authors.user: chen-yingfa
paper_authors._id: 66db3dfca46deac8798e3ca0
paper_authors.name: Weilin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:56:32.800Z
paper_authors._id: 64b8ff3d95bd42c770878042
paper_authors.avatarUrl: /avatars/564a4dccdf9e5b813a99979b0ef58183.svg
paper_authors.fullname: Weilin Zhao
paper_authors.type: user
paper_authors.user: Achazwl
paper_authors._id: 66db3dfca46deac8798e3ca1
paper_authors.name: Yuge Tu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:56:40.581Z
paper_authors._id: 637f3220e0d5fd3199579458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637f3220e0d5fd3199579458/uaPvnmGqRcScz9RgoUcVY.png
paper_authors.fullname: Yuge Tu
paper_authors.type: user
paper_authors.user: tuyuge
paper_authors._id: 66db3dfca46deac8798e3ca2
paper_authors.name: Zexuan Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:56:46.564Z
paper_authors._id: 6514dd906e0d30ed7e0df37e
paper_authors.avatarUrl: /avatars/893fb3fe2db1b684c2d9801d9f00aa57.svg
paper_authors.fullname: Zexuan Zhong
paper_authors.type: user
paper_authors.user: a3616001
paper_authors._id: 66db3dfca46deac8798e3ca3
paper_authors.name: Ao Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:57:03.318Z
paper_authors._id: 649a9133225ce03f1c82e0b2
paper_authors.avatarUrl: /avatars/f3702a46a0a310d0292dc690794e7cc9.svg
paper_authors.fullname: Ao Zhang
paper_authors.type: user
paper_authors.user: AoZhang
paper_authors._id: 66db3dfca46deac8798e3ca4
paper_authors.name: Chenglei Si
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:57:09.916Z
paper_authors._id: 6144c0aa8a3e54bb3ecef8ea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1631895645405-noauth.jpeg
paper_authors.fullname: Chenglei Si
paper_authors.type: user
paper_authors.user: CLS
paper_authors._id: 66db3dfca46deac8798e3ca5
paper_authors.name: Khai Hao Moo
paper_authors._id: 66db3dfca46deac8798e3ca6
paper_authors.name: Chenyang Zhao
paper_authors._id: 66db3dfca46deac8798e3ca7
paper_authors.name: Huimin Chen
paper_authors._id: 66db3dfca46deac8798e3ca8
paper_authors.name: Yankai Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:57:44.913Z
paper_authors._id: 657a651e1433ea7d44de6397
paper_authors.avatarUrl: /avatars/ccfc76f94595a38ff4a80f77c911eabf.svg
paper_authors.fullname: Yankai Lin
paper_authors.type: user
paper_authors.user: lyk423
paper_authors._id: 66db3dfca46deac8798e3ca9
paper_authors.name: Zhiyuan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:58:01.998Z
paper_authors._id: 6472f1f7485a7c8e1bcf507b
paper_authors.avatarUrl: /avatars/79dca137583875191ea883b193b630b7.svg
paper_authors.fullname: Zhiyuan Liu
paper_authors.type: user
paper_authors.user: zibuyu9
paper_authors._id: 66db3dfca46deac8798e3caa
paper_authors.name: Jingbo Shang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-09T07:58:13.783Z
paper_authors._id: 660655119e3555d648f6c6b5
paper_authors.avatarUrl: /avatars/ae1e2c97a08be39b77a9f1a5c2a718ef.svg
paper_authors.fullname: Jingbo Shang
paper_authors.type: user
paper_authors.user: shangjingbo
paper_authors._id: 66db3dfca46deac8798e3cab
paper_authors.name: Maosong Sun
paper_title: Configurable Foundation Models: Building LLMs from a Modular Perspective
paper_summary: Advancements in LLMs have recently unveiled challenges tied to computational
  efficiency and continual scalability due to their requirements of huge
  parameters, making the applications and evolution of these models on devices
  with limited computation resources and scenarios requiring various abilities
  increasingly cumbersome. Inspired by modularity within the human brain, there
  is a growing tendency to decompose LLMs into numerous functional modules,
  allowing for inference with part of modules and dynamic assembly of modules to
  tackle complex tasks, such as mixture-of-experts. To highlight the inherent
  efficiency and composability of the modular approach, we coin the term brick to
  represent each functional module, designating the modularized structure as
  configurable foundation models. In this paper, we offer a comprehensive
  overview and investigation of the construction, utilization, and limitation of
  configurable foundation models. We first formalize modules into emergent bricks
  - functional neuron partitions that emerge during the pre-training phase, and
  customized bricks - bricks constructed via additional post-training to improve
  the capabilities and knowledge of LLMs. Based on diverse functional bricks, we
  further present four brick-oriented operations: retrieval and routing, merging,
  updating, and growing. These operations allow for dynamic configuration of LLMs
  based on instructions to handle complex tasks. To verify our perspective, we
  conduct an empirical analysis on widely-used LLMs. We find that the FFN layers
  follow modular patterns with functional specialization of neurons and
  functional neuron partitions. Finally, we highlight several open issues and
  directions for future research. Overall, this paper aims to offer a fresh
  modular perspective on existing LLM research and inspire the future creation of
  more efficient and scalable foundational models.
paper_ai_summary: The paper explores a modular approach for large language models (LLMs) by decomposing them into functional modules (bricks) to enhance computational efficiency and scalability.
paper_ai_keywords: LLMs computational efficiency continual scalability parameters modularizing functional modules mixture-of-experts configurable foundation models emergent bricks customized bricks retrieval and routing merging updating growing FFN layers functional specialization functional neuron partitions