title: Frac-Connections: Fractional Extension of Hyper-Connections
summary: Residual connections are central to modern deep learning architectures,
  enabling the training of very deep networks by mitigating gradient vanishing.
  Hyper-Connections recently generalized residual connections by introducing
  multiple connection strengths at different depths, thereby addressing the
  seesaw effect between gradient vanishing and representation collapse. However,
  Hyper-Connections increase memory access costs by expanding the width of hidden
  states. In this paper, we propose Frac-Connections, a novel approach that
  divides hidden states into multiple parts rather than expanding their width.
  Frac-Connections retain partial benefits of Hyper-Connections while reducing
  memory consumption. To validate their effectiveness, we conduct large-scale
  experiments on language tasks, with the largest being a 7B MoE model trained on
  up to 3T tokens, demonstrating that Frac-Connections significantly outperform
  residual connections.
paper_authors._id: 67da200db41738a058666623
paper_authors.name: Defa Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:32:01.715Z
paper_authors._id: 667505f4361b960c79e35486
paper_authors.avatarUrl: /avatars/d352639c520075220f6abaae23c39376.svg
paper_authors.fullname: Defa Zhu
paper_authors.type: user
paper_authors.user: mathfinder
paper_authors._id: 67da200db41738a058666624
paper_authors.name: Hongzhi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:32:07.284Z
paper_authors._id: 66defe949fda73ea123ede1c
paper_authors.avatarUrl: /avatars/52bda14a533fdd13436bcf7c65a38fb3.svg
paper_authors.fullname: huang
paper_authors.type: user
paper_authors.user: hongzhihuang
paper_authors._id: 67da200db41738a058666625
paper_authors.name: Jundong Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:32:22.854Z
paper_authors._id: 642ae9f4d651bae3c11ebad1
paper_authors.avatarUrl: /avatars/d0f9253da54e148c152710923d8d3804.svg
paper_authors.fullname: Zhou Jundong
paper_authors.type: user
paper_authors.user: ulrical
paper_authors._id: 67da200db41738a058666626
paper_authors.name: Zihao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:45:07.454Z
paper_authors._id: 65a62085576772f531e13856
paper_authors.avatarUrl: /avatars/72c67a60422e333ea4e323f7480ae0b7.svg
paper_authors.fullname: Huang Zihao
paper_authors.type: user
paper_authors.user: FetchFortune
paper_authors._id: 67da200db41738a058666627
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-19T09:45:05.382Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 67da200db41738a058666628
paper_authors.name: Banggu Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:32:34.670Z
paper_authors._id: 64a3b0fa565496b629879293
paper_authors.avatarUrl: /avatars/9d5de6cc4a01052ea971701f72bd3489.svg
paper_authors.fullname: wubanggu
paper_authors.type: user
paper_authors.user: banggu
paper_authors._id: 67da200db41738a058666629
paper_authors.name: Qiyang Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-19T13:32:46.730Z
paper_authors._id: 645507b89d37c3fb33279fe3
paper_authors.avatarUrl: /avatars/ec1122137c49204ab968182d1f726c35.svg
paper_authors.fullname: min
paper_authors.type: user
paper_authors.user: qiyang-attn
paper_authors._id: 67da200db41738a05866662a
paper_authors.name: Xun Zhou
paper_title: Frac-Connections: Fractional Extension of Hyper-Connections
paper_summary: Residual connections are central to modern deep learning architectures,
  enabling the training of very deep networks by mitigating gradient vanishing.
  Hyper-Connections recently generalized residual connections by introducing
  multiple connection strengths at different depths, thereby addressing the
  seesaw effect between gradient vanishing and representation collapse. However,
  Hyper-Connections increase memory access costs by expanding the width of hidden
  states. In this paper, we propose Frac-Connections, a novel approach that
  divides hidden states into multiple parts rather than expanding their width.
  Frac-Connections retain partial benefits of Hyper-Connections while reducing
  memory consumption. To validate their effectiveness, we conduct large-scale
  experiments on language tasks, with the largest being a 7B MoE model trained on
  up to 3T tokens, demonstrating that Frac-Connections significantly outperform
  residual connections.
paper_ai_summary: Frac-Connections, a technique that divides hidden states, outperform traditional residual connections in deep learning by balancing memory consumption and performance, validated through experiments on large language models.
paper_ai_keywords: residual connections gradient vanishing Hyper-Connections representation collapse Frac-Connections hidden states memory consumption large-scale experiments language tasks 7B MoE model 3T tokens