title: Ultra-Sparse Memory Network
summary: It is widely acknowledged that the performance of Transformer models is
  exponentially related to their number of parameters and computational
  complexity. While approaches like Mixture of Experts (MoE) decouple parameter
  count from computational complexity, they still face challenges in inference
  due to high memory access costs. This work introduces UltraMem, incorporating
  large-scale, ultra-sparse memory layer to address these limitations. Our
  approach significantly reduces inference latency while maintaining model
  performance. We also investigate the scaling laws of this new architecture,
  demonstrating that it not only exhibits favorable scaling properties but
  outperforms traditional models. In our experiments, we train networks with up
  to 20 million memory slots. The results show that our method achieves
  state-of-the-art inference speed and model performance within a given
  computational budget.
paper_authors._id: 673ffaea27b90045bdea1841
paper_authors.name: Zihao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:01:54.226Z
paper_authors._id: 65a62085576772f531e13856
paper_authors.avatarUrl: /avatars/72c67a60422e333ea4e323f7480ae0b7.svg
paper_authors.fullname: Huang Zihao
paper_authors.type: user
paper_authors.user: FetchFortune
paper_authors._id: 673ffaea27b90045bdea1842
paper_authors.name: Qiyang Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:12:01.276Z
paper_authors._id: 645507b89d37c3fb33279fe3
paper_authors.avatarUrl: /avatars/ec1122137c49204ab968182d1f726c35.svg
paper_authors.fullname: min
paper_authors.type: user
paper_authors.user: qiyang-attn
paper_authors._id: 673ffaea27b90045bdea1843
paper_authors.name: Hongzhi Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:01:50.933Z
paper_authors._id: 66fbbed8a0154e0b6498293d
paper_authors.avatarUrl: /avatars/5c4202f13cc5af6424154fae293fad52.svg
paper_authors.fullname: Huang Hongzhi
paper_authors.type: user
paper_authors.user: xyzed
paper_authors._id: 673ffaea27b90045bdea1844
paper_authors.name: Defa Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:12:08.908Z
paper_authors._id: 667505f4361b960c79e35486
paper_authors.avatarUrl: /avatars/d352639c520075220f6abaae23c39376.svg
paper_authors.fullname: Defa Zhu
paper_authors.type: user
paper_authors.user: mathfinder
paper_authors._id: 673ffaea27b90045bdea1845
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:01:52.553Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 673ffaea27b90045bdea1846
paper_authors.name: Ran Guo
paper_authors._id: 673ffaea27b90045bdea1847
paper_authors.name: Xun Zhou
paper_title: Ultra-Sparse Memory Network
paper_summary: It is widely acknowledged that the performance of Transformer models is
  exponentially related to their number of parameters and computational
  complexity. While approaches like Mixture of Experts (MoE) decouple parameter
  count from computational complexity, they still face challenges in inference
  due to high memory access costs. This work introduces UltraMem, incorporating
  large-scale, ultra-sparse memory layer to address these limitations. Our
  approach significantly reduces inference latency while maintaining model
  performance. We also investigate the scaling laws of this new architecture,
  demonstrating that it not only exhibits favorable scaling properties but
  outperforms traditional models. In our experiments, we train networks with up
  to 20 million memory slots. The results show that our method achieves
  state-of-the-art inference speed and model performance within a given
  computational budget.
paper_ai_summary: UltraMem introduces a large-scale, ultra-sparse memory layer to Transformer models, reducing inference latency and improving performance.
paper_ai_keywords: Mixture of Experts Transformer models UltraMem ultra-sparse memory layer inference latency model performance scaling laws