title: Revisiting Residual Connections: Orthogonal Updates for Stable and
    Efficient Deep Networks
summary: Residual connections are pivotal for deep neural networks, enabling greater
  depth by mitigating vanishing gradients. However, in standard residual updates,
  the module's output is directly added to the input stream. This can lead to
  updates that predominantly reinforce or modulate the existing stream direction,
  potentially underutilizing the module's capacity for learning entirely novel
  features. In this work, we introduce Orthogonal Residual Update: we decompose
  the module's output relative to the input stream and add only the component
  orthogonal to this stream. This design aims to guide modules to contribute
  primarily new representational directions, fostering richer feature learning
  while promoting more efficient training. We demonstrate that our orthogonal
  update strategy improves generalization accuracy and training stability across
  diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs,
  TinyImageNet, ImageNet-1k), achieving, for instance, a +4.3\%p top-1 accuracy
  gain for ViT-B on ImageNet-1k.
paper_authors._id: 6833ebdb142b0e50399413d3
paper_authors.name: Giyeong Oh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:02.174Z
paper_authors._id: 63d93667255ef6add20f9272
paper_authors.avatarUrl: /avatars/99a3aeadcc81ef85164cdfb6ab186b17.svg
paper_authors.fullname: Giyeong Oh
paper_authors.type: user
paper_authors.user: BootsofLagrangian
paper_authors._id: 6833ebdb142b0e50399413d4
paper_authors.name: Woohyun Cho
paper_authors._id: 6833ebdb142b0e50399413d5
paper_authors.name: Siyeol Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:04.269Z
paper_authors._id: 67021743e4d49b157afd8260
paper_authors.avatarUrl: /avatars/2a22a18cd45f6d115e8a3a5d1e477dcb.svg
paper_authors.fullname: Siyeol Kim
paper_authors.type: user
paper_authors.user: siyeolkim
paper_authors._id: 6833ebdb142b0e50399413d6
paper_authors.name: Suhwan Choi
paper_authors._id: 6833ebdb142b0e50399413d7
paper_authors.name: Younjae Yu
paper_title: Revisiting Residual Connections: Orthogonal Updates for Stable and
    Efficient Deep Networks
paper_summary: Residual connections are pivotal for deep neural networks, enabling greater
  depth by mitigating vanishing gradients. However, in standard residual updates,
  the module's output is directly added to the input stream. This can lead to
  updates that predominantly reinforce or modulate the existing stream direction,
  potentially underutilizing the module's capacity for learning entirely novel
  features. In this work, we introduce Orthogonal Residual Update: we decompose
  the module's output relative to the input stream and add only the component
  orthogonal to this stream. This design aims to guide modules to contribute
  primarily new representational directions, fostering richer feature learning
  while promoting more efficient training. We demonstrate that our orthogonal
  update strategy improves generalization accuracy and training stability across
  diverse architectures (ResNetV2, Vision Transformers) and datasets (CIFARs,
  TinyImageNet, ImageNet-1k), achieving, for instance, a +4.3\%p top-1 accuracy
  gain for ViT-B on ImageNet-1k.
paper_ai_summary: Orthogonal Residual Updates enhance feature learning and training stability by decomposing module outputs to contribute primarily novel features.
paper_ai_keywords: residual connections vanishing gradients orthogonal update ResNetV2 Vision Transformers CIFARs TinyImageNet ImageNet-1k top-1 accuracy