Answers.Body: <p>Lateral connection in NN simply means that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) then lateral connection implies that unit 1 may be connected to unit 2.</p>
  <p>So now the activation of unit 2 is not only dependent on incoming inputs but also on unit 1. The weight assigned to a connection between unit 2 and unit 1 needs to be learned as part of backprop.</p>
  <p>Please refer to the below image for more details :</p>
  <p><a href="https://i.sstatic.net/U8K0H.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/U8K0H.png" alt="enter image description here" /></a></p>
  <blockquote>
  <p>Lateral connections in a feed-forward architecture. Inputs and hidden layer neurons are fully connected as are hidden layer and output layer neurons. Neuron j in the hidden layer also receives the net input of neuron (j - 1) in the hidden layer through a lateral connection</p>
  </blockquote>
  <p>Ref: <a href="https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0" rel="nofollow noreferrer">https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0</a></p>
  <p><strong>Lateral Connection can occur between units in layers and can be applied to feed forward network</strong></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-06T12:57:39.997Z
Answers.LastActivityDate: 2022-05-22T11:38:46.823Z
Answers.LastEditDate: 2022-05-22T11:38:46.823Z
Answers.UserDisplayName: Ashwiniku918
Body: <p>A class of CNN is popular due to the implementation of residual connections.</p>
  <p>We can use both terms &quot;residual connections&quot; and &quot;skip connections&quot; interchangeably <a href="https://stats.stackexchange.com/a/321112/42883">as they refer to the same.</a></p>
  <blockquote>
  <p>Residual connections are the same thing as 'skip connections'. They
  are used to allow gradients to flow through a network directly,
  without passing through non-linear activation functions. Non-linear
  activation functions, by nature of being non-linear, cause the
  gradients to explode or vanish (depending on the weights).</p>
  </blockquote>
  <p>While studying some models that are used in codes of some deep learning projects, I came across the word &quot;lateral connections&quot;.</p>
  <p>I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.</p>
  <p>Am I true? If not, when can I call a connection lateral?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-01-06T13:00:13.113Z
Comments.Text: When you say "While studying some models that are used in codes of some deep learning projects", it may be a good idea to provide the source where you read about this term.
Comments.UserDisplayName: nbro
Tags: <neural-networks><terminology>
Title: What is meant by "lateral connection" in the context of neural networks?
UserDisplayName: hanugm
ThreadText: # What is meant by "lateral connection" in the context of neural networks?
  
  (Asked by: username_0 on 2022-01-06)
  
  A class of CNN is popular due to the implementation of residual connections.
  
  We can use both terms "residual connections" and "skip connections" interchangeably [as they refer to the same.](https://stats.stackexchange.com/a/321112/42883)
  
  > Residual connections are the same thing as 'skip connections'. They
  > are used to allow gradients to flow through a network directly,
  > without passing through non-linear activation functions. Non-linear
  > activation functions, by nature of being non-linear, cause the
  > gradients to explode or vanish (depending on the weights).
  
  While studying some models that are used in codes of some deep learning projects, I came across the word "lateral connections".
  
  I guess that lateral connections stand for the connections that are not present in the traditional feed-forward neural networks of any kind. Lateral connections may be from any layer to any other layer.
  
  Am I true? If not, when can I call a connection lateral?
  
  **Tags:** neural-networks, terminology
  
  > **Comments:**
  > **username_1**: When you say "While studying some models that are used in codes of some deep learning projects", it may be a good idea to provide the source where you read about this term.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-01-06. Score: 3 | Accepted answer
  
  Lateral connection in NN simply means that units in a hidden layer are connected with one another. Suppose we have a hidden layer Hi which has 10 from (1,2,....10) then lateral connection implies that unit 1 may be connected to unit 2.
  
  So now the activation of unit 2 is not only dependent on incoming inputs but also on unit 1. The weight assigned to a connection between unit 2 and unit 1 needs to be learned as part of backprop.
  
  Please refer to the below image for more details :
  
  [![enter image description here](https://i.sstatic.net/U8K0H.png)](https://i.sstatic.net/U8K0H.png)
  
  > Lateral connections in a feed-forward architecture. Inputs and hidden layer neurons are fully connected as are hidden layer and output layer neurons. Neuron j in the hidden layer also receives the net input of neuron (j - 1) in the hidden layer through a lateral connection
  
  Ref: <https://www.semanticscholar.org/paper/On-lateral-connections-in-feed-forward-neural-Kothari-Agyepong/167d7c20dcf37fba0388b0365d9bd42ba8b2ecde/figure/0>
  
  **Lateral Connection can occur between units in layers and can be applied to feed forward network**
  
  
  ---
  
  