Ultra-Sparse Memory Network
It is widely acknowledged that the performance of Transformer models is
exponentially related to their number of parameters and computational
complexity. While approaches like Mixture of Experts (MoE) decouple parameter
count from computational complexity, they still face challenges in inference
due to high memory access costs. This work introduces UltraMem, incorporating
large-scale, ultra-sparse memory layer to address these limitations. Our
approach significantly reduces inference latency while maintaining model
performance. We also investigate the scaling laws of this new architecture,
demonstrating that it not only exhibits favorable scaling properties but
outperforms traditional models. In our experiments, we train networks with up
to 20 million memory slots. The results show that our method achieves
state-of-the-art inference speed and model performance within a given
computational budget.
673ffaea27b90045bdea1841
Zihao Huang
claimed_verified
2024-11-22T10:01:54.226Z
65a62085576772f531e13856
/avatars/72c67a60422e333ea4e323f7480ae0b7.svg
Huang Zihao
user
FetchFortune
673ffaea27b90045bdea1842
Qiyang Min
admin_assigned
2024-11-22T13:12:01.276Z
645507b89d37c3fb33279fe3
/avatars/ec1122137c49204ab968182d1f726c35.svg
min
user
qiyang-attn
673ffaea27b90045bdea1843
Hongzhi Huang
claimed_verified
2024-11-22T10:01:50.933Z
66fbbed8a0154e0b6498293d
/avatars/5c4202f13cc5af6424154fae293fad52.svg
Huang Hongzhi
user
xyzed
673ffaea27b90045bdea1844
Defa Zhu
admin_assigned
2024-11-22T13:12:08.908Z
667505f4361b960c79e35486
/avatars/d352639c520075220f6abaae23c39376.svg
Defa Zhu
user
mathfinder
673ffaea27b90045bdea1845
Yutao Zeng
claimed_verified
2024-11-22T10:01:52.553Z
6371128eafbe42caa5a5222b
/avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
Yutao Zeng
user
Taoer
673ffaea27b90045bdea1846
Ran Guo
673ffaea27b90045bdea1847
Xun Zhou
Ultra-Sparse Memory Network
It is widely acknowledged that the performance of Transformer models is
exponentially related to their number of parameters and computational
complexity. While approaches like Mixture of Experts (MoE) decouple parameter
count from computational complexity, they still face challenges in inference
due to high memory access costs. This work introduces UltraMem, incorporating
large-scale, ultra-sparse memory layer to address these limitations. Our
approach significantly reduces inference latency while maintaining model
performance. We also investigate the scaling laws of this new architecture,
demonstrating that it not only exhibits favorable scaling properties but
outperforms traditional models. In our experiments, we train networks with up
to 20 million memory slots. The results show that our method achieves
state-of-the-art inference speed and model performance within a given
computational budget.
UltraMem introduces a large-scale, ultra-sparse memory layer to Transformer models, reducing inference latency and improving performance.
Mixture of Experts Transformer models UltraMem ultra-sparse memory layer inference latency model performance scaling laws