title: Hyper-Connections
summary: We present hyper-connections, a simple yet effective method that can serve as
  an alternative to residual connections. This approach specifically addresses
  common drawbacks observed in residual connection variants, such as the seesaw
  effect between gradient vanishing and representation collapse. Theoretically,
  hyper-connections allow the network to adjust the strength of connections
  between features at different depths and dynamically rearrange layers. We
  conduct experiments focusing on the pre-training of large language models,
  including dense and sparse models, where hyper-connections show significant
  performance improvements over residual connections. Additional experiments
  conducted on vision tasks also demonstrate similar improvements. We anticipate
  that this method will be broadly applicable and beneficial across a wide range
  of AI problems.
paper_authors._id: 66fb8def1c69086d311e8202
paper_authors.name: Defa Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:40:16.606Z
paper_authors._id: 667505f4361b960c79e35486
paper_authors.avatarUrl: /avatars/d352639c520075220f6abaae23c39376.svg
paper_authors.fullname: Defa Zhu
paper_authors.type: user
paper_authors.user: mathfinder
paper_authors._id: 66fb8def1c69086d311e8203
paper_authors.name: Hongzhi Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:40:21.956Z
paper_authors._id: 66defe949fda73ea123ede1c
paper_authors.avatarUrl: /avatars/52bda14a533fdd13436bcf7c65a38fb3.svg
paper_authors.fullname: huang
paper_authors.type: user
paper_authors.user: hongzhihuang
paper_authors._id: 66fb8def1c69086d311e8204
paper_authors.name: Zihao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T09:10:58.219Z
paper_authors._id: 65a62085576772f531e13856
paper_authors.avatarUrl: /avatars/72c67a60422e333ea4e323f7480ae0b7.svg
paper_authors.fullname: Huang Zihao
paper_authors.type: user
paper_authors.user: FetchFortune
paper_authors._id: 66fb8def1c69086d311e8205
paper_authors.name: Yutao Zeng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-01T07:16:36.484Z
paper_authors._id: 6371128eafbe42caa5a5222b
paper_authors.avatarUrl: /avatars/c3b2ab35949c38aa3dfb2657a1300aac.svg
paper_authors.fullname: Yutao Zeng
paper_authors.type: user
paper_authors.user: Taoer
paper_authors._id: 66fb8def1c69086d311e8206
paper_authors.name: Yunyao Mao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:40:53.027Z
paper_authors._id: 662b1e3b637672f56f81d4b7
paper_authors.avatarUrl: /avatars/f5367089a6c69b3538bf8c9f8b411864.svg
paper_authors.fullname: Yunyao Mao
paper_authors.type: user
paper_authors.user: YunyaoMao
paper_authors._id: 66fb8def1c69086d311e8207
paper_authors.name: Banggu Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:41:05.200Z
paper_authors._id: 64a3b0fa565496b629879293
paper_authors.avatarUrl: /avatars/9d5de6cc4a01052ea971701f72bd3489.svg
paper_authors.fullname: wubanggu
paper_authors.type: user
paper_authors.user: banggu
paper_authors._id: 66fb8def1c69086d311e8208
paper_authors.name: Qiyang Min
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:41:16.184Z
paper_authors._id: 645507b89d37c3fb33279fe3
paper_authors.avatarUrl: /avatars/ec1122137c49204ab968182d1f726c35.svg
paper_authors.fullname: min
paper_authors.type: user
paper_authors.user: qiyang-attn
paper_authors._id: 66fb8def1c69086d311e8209
paper_authors.name: Xun Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-01T08:41:43.253Z
paper_authors._id: 62533db4a06ec75172eeabe7
paper_authors.avatarUrl: /avatars/b1a4dad90afae5c00df97233a97777db.svg
paper_authors.fullname: xunzhou
paper_authors.type: user
paper_authors.user: xunzhou
paper_title: Hyper-Connections
paper_summary: We present hyper-connections, a simple yet effective method that can serve as
  an alternative to residual connections. This approach specifically addresses
  common drawbacks observed in residual connection variants, such as the seesaw
  effect between gradient vanishing and representation collapse. Theoretically,
  hyper-connections allow the network to adjust the strength of connections
  between features at different depths and dynamically rearrange layers. We
  conduct experiments focusing on the pre-training of large language models,
  including dense and sparse models, where hyper-connections show significant
  performance improvements over residual connections. Additional experiments
  conducted on vision tasks also demonstrate similar improvements. We anticipate
  that this method will be broadly applicable and beneficial across a wide range
  of AI problems.
paper_ai_keywords: hyper-connections residual connections seesaw effect gradient vanishing representation collapse pre-training large language models dense models sparse models dynamic rearrangement of layers