title: Chain-of-Model Learning for Language Model
summary: In this paper, we propose a novel learning paradigm, termed Chain-of-Model
  (CoM), which incorporates the causal relationship into the hidden states of
  each layer as a chain style, thereby introducing great scaling efficiency in
  model training and inference flexibility in deployment. We introduce the
  concept of Chain-of-Representation (CoR), which formulates the hidden states at
  each layer as a combination of multiple sub-representations (i.e., chains) at
  the hidden dimension level. In each layer, each chain from the output
  representations can only view all of its preceding chains in the input
  representations. Consequently, the model built upon CoM framework can
  progressively scale up the model size by increasing the chains based on the
  previous models (i.e., chains), and offer multiple sub-models at varying sizes
  for elastic inference by using different chain numbers. Based on this
  principle, we devise Chain-of-Language-Model (CoLM), which incorporates the
  idea of CoM into each layer of Transformer architecture. Based on CoLM, we
  further introduce CoLM-Air by introducing a KV sharing mechanism, that computes
  all keys and values within the first chain and then shares across all chains.
  This design demonstrates additional extensibility, such as enabling seamless LM
  switching, prefilling acceleration and so on. Experimental results demonstrate
  our CoLM family can achieve comparable performance to the standard Transformer,
  while simultaneously enabling greater flexiblity, such as progressive scaling
  to improve training efficiency and offer multiple varying model sizes for
  elastic inference, paving a a new way toward building language models. Our code
  will be released in the future at: https://github.com/microsoft/CoLM.
paper_authors._id: 682bf779fdfa3c5de0eb1e02
paper_authors.name: Kaitao Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T08:07:27.158Z
paper_authors._id: 5fc0b2b61160c47d1d438568
paper_authors.avatarUrl: /avatars/b355912b0ec683e73f21c8d36620e146.svg
paper_authors.fullname: Kaitao Song
paper_authors.type: user
paper_authors.user: KaitaoSong
paper_authors._id: 682bf779fdfa3c5de0eb1e03
paper_authors.name: Xiaohua Wang
paper_authors._id: 682bf779fdfa3c5de0eb1e04
paper_authors.name: Xu Tan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:20:36.359Z
paper_authors._id: 5f1040b6e9d71719e3be71d2
paper_authors.avatarUrl: /avatars/a2f28940236ae625ed3810ad62e343ff.svg
paper_authors.fullname: Xu Tan
paper_authors.type: user
paper_authors.user: xutan
paper_authors._id: 682bf779fdfa3c5de0eb1e05
paper_authors.name: Huiqiang Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:08:04.470Z
paper_authors._id: 6278bd42541f3d2dfa77ea70
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
paper_authors.fullname: Huiqiang Jiang
paper_authors.type: user
paper_authors.user: iofu728
paper_authors._id: 682bf779fdfa3c5de0eb1e06
paper_authors.name: Chengruidong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:08:14.328Z
paper_authors._id: 64646896884f2e3e1ced3cd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: Chengruidong
paper_authors._id: 682bf779fdfa3c5de0eb1e07
paper_authors.name: Yongliang Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T07:23:50.224Z
paper_authors._id: 5e1058e9fcf41d740b69966d
paper_authors.avatarUrl: /avatars/ce74839ba871f2b54313a670a233ba82.svg
paper_authors.fullname: Yongliang Shen
paper_authors.type: user
paper_authors.user: tricktreat
paper_authors._id: 682bf779fdfa3c5de0eb1e08
paper_authors.name: Cen LU
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-20T19:29:34.585Z
paper_authors._id: 6699d46d990749deca71598b
paper_authors.avatarUrl: /avatars/5f2bd1447d40a4ae03f35734cbc4228c.svg
paper_authors.fullname: Cen
paper_authors.type: user
paper_authors.user: RienV
paper_authors._id: 682bf779fdfa3c5de0eb1e09
paper_authors.name: Zihao Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:22:54.176Z
paper_authors._id: 65d41143406bdae630826d87
paper_authors.avatarUrl: /avatars/ed8259ed64a13b1ed43c004b741e1401.svg
paper_authors.fullname: Zihao Li
paper_authors.type: user
paper_authors.user: Violet24K
paper_authors._id: 682bf779fdfa3c5de0eb1e0a
paper_authors.name: Zifan Song
paper_authors._id: 682bf779fdfa3c5de0eb1e0b
paper_authors.name: Caihua Shan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:08:55.127Z
paper_authors._id: 66beeca13ae330ae8b63a0c9
paper_authors.avatarUrl: /avatars/09c8341beb8998e4506cef09e3481e77.svg
paper_authors.fullname: SHAN CAIHUA
paper_authors.type: user
paper_authors.user: sxdtgg
paper_authors._id: 682bf779fdfa3c5de0eb1e0c
paper_authors.name: Yansen Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:09:02.859Z
paper_authors._id: 678e0bd1ef7630e73c4ad508
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/LlHuTu9VuUJl3jaJzuWly.png
paper_authors.fullname: Yansen Wang
paper_authors.type: user
paper_authors.user: victorywys
paper_authors._id: 682bf779fdfa3c5de0eb1e0d
paper_authors.name: Kan Ren
paper_authors._id: 682bf779fdfa3c5de0eb1e0e
paper_authors.name: Xiaoqing Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-20T08:09:16.810Z
paper_authors._id: 680331764422d7ba43db26cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sDKtCDf71fAljNC_SAq_C.png
paper_authors.fullname: zheng xiaoqing
paper_authors.type: user
paper_authors.user: Qu1zas
paper_authors._id: 682bf779fdfa3c5de0eb1e0f
paper_authors.name: Tao Qin
paper_authors._id: 682bf779fdfa3c5de0eb1e10
paper_authors.name: Yuqing Yang
paper_authors._id: 682bf779fdfa3c5de0eb1e11
paper_authors.name: Dongsheng Li
paper_authors._id: 682bf779fdfa3c5de0eb1e12
paper_authors.name: Lili Qiu
paper_title: Chain-of-Model Learning for Language Model
paper_summary: In this paper, we propose a novel learning paradigm, termed Chain-of-Model
  (CoM), which incorporates the causal relationship into the hidden states of
  each layer as a chain style, thereby introducing great scaling efficiency in
  model training and inference flexibility in deployment. We introduce the
  concept of Chain-of-Representation (CoR), which formulates the hidden states at
  each layer as a combination of multiple sub-representations (i.e., chains) at
  the hidden dimension level. In each layer, each chain from the output
  representations can only view all of its preceding chains in the input
  representations. Consequently, the model built upon CoM framework can
  progressively scale up the model size by increasing the chains based on the
  previous models (i.e., chains), and offer multiple sub-models at varying sizes
  for elastic inference by using different chain numbers. Based on this
  principle, we devise Chain-of-Language-Model (CoLM), which incorporates the
  idea of CoM into each layer of Transformer architecture. Based on CoLM, we
  further introduce CoLM-Air by introducing a KV sharing mechanism, that computes
  all keys and values within the first chain and then shares across all chains.
  This design demonstrates additional extensibility, such as enabling seamless LM
  switching, prefilling acceleration and so on. Experimental results demonstrate
  our CoLM family can achieve comparable performance to the standard Transformer,
  while simultaneously enabling greater flexiblity, such as progressive scaling
  to improve training efficiency and offer multiple varying model sizes for
  elastic inference, paving a a new way toward building language models. Our code
  will be released in the future at: https://github.com/microsoft/CoLM.
paper_ai_summary: A novel Chain-of-Model framework introduces hierarchical hidden state chains in Transformers to improve scaling efficiency and inference flexibility for language models.
paper_ai_keywords: Chain-of-Model (CoM) Chain-of-Representation (CoR) sub-representations input representations output representations Chain-of-Language-Model (CoLM) KV sharing mechanism prefilling acceleration Transformer architecture elastic inference progressive scaling