Chain-of-Model Learning for Language Model
In this paper, we propose a novel learning paradigm, termed Chain-of-Model
(CoM), which incorporates the causal relationship into the hidden states of
each layer as a chain style, thereby introducing great scaling efficiency in
model training and inference flexibility in deployment. We introduce the
concept of Chain-of-Representation (CoR), which formulates the hidden states at
each layer as a combination of multiple sub-representations (i.e., chains) at
the hidden dimension level. In each layer, each chain from the output
representations can only view all of its preceding chains in the input
representations. Consequently, the model built upon CoM framework can
progressively scale up the model size by increasing the chains based on the
previous models (i.e., chains), and offer multiple sub-models at varying sizes
for elastic inference by using different chain numbers. Based on this
principle, we devise Chain-of-Language-Model (CoLM), which incorporates the
idea of CoM into each layer of Transformer architecture. Based on CoLM, we
further introduce CoLM-Air by introducing a KV sharing mechanism, that computes
all keys and values within the first chain and then shares across all chains.
This design demonstrates additional extensibility, such as enabling seamless LM
switching, prefilling acceleration and so on. Experimental results demonstrate
our CoLM family can achieve comparable performance to the standard Transformer,
while simultaneously enabling greater flexiblity, such as progressive scaling
to improve training efficiency and offer multiple varying model sizes for
elastic inference, paving a a new way toward building language models. Our code
will be released in the future at: https://github.com/microsoft/CoLM.
682bf779fdfa3c5de0eb1e02
Kaitao Song
claimed_verified
2025-05-20T08:07:27.158Z
5fc0b2b61160c47d1d438568
/avatars/b355912b0ec683e73f21c8d36620e146.svg
Kaitao Song
user
KaitaoSong
682bf779fdfa3c5de0eb1e03
Xiaohua Wang
682bf779fdfa3c5de0eb1e04
Xu Tan
claimed_verified
2025-05-20T07:20:36.359Z
5f1040b6e9d71719e3be71d2
/avatars/a2f28940236ae625ed3810ad62e343ff.svg
Xu Tan
user
xutan
682bf779fdfa3c5de0eb1e05
Huiqiang Jiang
admin_assigned
2025-05-20T08:08:04.470Z
6278bd42541f3d2dfa77ea70
https://cdn-avatars.huggingface.co/v1/production/uploads/6278bd42541f3d2dfa77ea70/ejn49eapnB3UXQckAYdTd.jpeg
Huiqiang Jiang
user
iofu728
682bf779fdfa3c5de0eb1e06
Chengruidong Zhang
admin_assigned
2025-05-20T08:08:14.328Z
64646896884f2e3e1ced3cd5
https://cdn-avatars.huggingface.co/v1/production/uploads/64646896884f2e3e1ced3cd5/86-t8V8LGMNaPQRXnADiD.png
Zhang
user
Chengruidong
682bf779fdfa3c5de0eb1e07
Yongliang Shen
claimed_verified
2025-05-20T07:23:50.224Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
682bf779fdfa3c5de0eb1e08
Cen LU
claimed_verified
2025-05-20T19:29:34.585Z
6699d46d990749deca71598b
/avatars/5f2bd1447d40a4ae03f35734cbc4228c.svg
Cen
user
RienV
682bf779fdfa3c5de0eb1e09
Zihao Li
claimed_verified
2025-05-26T08:22:54.176Z
65d41143406bdae630826d87
/avatars/ed8259ed64a13b1ed43c004b741e1401.svg
Zihao Li
user
Violet24K
682bf779fdfa3c5de0eb1e0a
Zifan Song
682bf779fdfa3c5de0eb1e0b
Caihua Shan
admin_assigned
2025-05-20T08:08:55.127Z
66beeca13ae330ae8b63a0c9
/avatars/09c8341beb8998e4506cef09e3481e77.svg
SHAN CAIHUA
user
sxdtgg
682bf779fdfa3c5de0eb1e0c
Yansen Wang
admin_assigned
2025-05-20T08:09:02.859Z
678e0bd1ef7630e73c4ad508
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/LlHuTu9VuUJl3jaJzuWly.png
Yansen Wang
user
victorywys
682bf779fdfa3c5de0eb1e0d
Kan Ren
682bf779fdfa3c5de0eb1e0e
Xiaoqing Zheng
admin_assigned
2025-05-20T08:09:16.810Z
680331764422d7ba43db26cc
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sDKtCDf71fAljNC_SAq_C.png
zheng xiaoqing
user
Qu1zas
682bf779fdfa3c5de0eb1e0f
Tao Qin
682bf779fdfa3c5de0eb1e10
Yuqing Yang
682bf779fdfa3c5de0eb1e11
Dongsheng Li
682bf779fdfa3c5de0eb1e12
Lili Qiu
Chain-of-Model Learning for Language Model
In this paper, we propose a novel learning paradigm, termed Chain-of-Model
(CoM), which incorporates the causal relationship into the hidden states of
each layer as a chain style, thereby introducing great scaling efficiency in
model training and inference flexibility in deployment. We introduce the
concept of Chain-of-Representation (CoR), which formulates the hidden states at
each layer as a combination of multiple sub-representations (i.e., chains) at
the hidden dimension level. In each layer, each chain from the output
representations can only view all of its preceding chains in the input
representations. Consequently, the model built upon CoM framework can
progressively scale up the model size by increasing the chains based on the
previous models (i.e., chains), and offer multiple sub-models at varying sizes
for elastic inference by using different chain numbers. Based on this
principle, we devise Chain-of-Language-Model (CoLM), which incorporates the
idea of CoM into each layer of Transformer architecture. Based on CoLM, we
further introduce CoLM-Air by introducing a KV sharing mechanism, that computes
all keys and values within the first chain and then shares across all chains.
This design demonstrates additional extensibility, such as enabling seamless LM
switching, prefilling acceleration and so on. Experimental results demonstrate
our CoLM family can achieve comparable performance to the standard Transformer,
while simultaneously enabling greater flexiblity, such as progressive scaling
to improve training efficiency and offer multiple varying model sizes for
elastic inference, paving a a new way toward building language models. Our code
will be released in the future at: https://github.com/microsoft/CoLM.
A novel Chain-of-Model framework introduces hierarchical hidden state chains in Transformers to improve scaling efficiency and inference flexibility for language models.
Chain-of-Model (CoM) Chain-of-Representation (CoR) sub-representations input representations output representations Chain-of-Language-Model (CoLM) KV sharing mechanism prefilling acceleration Transformer architecture elastic inference progressive scaling