Answers.Body: <p>According to <a href="https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004128" rel="nofollow noreferrer">research I found</a>, they show that it helps the network generalise and avoid catastrophic forgeting, but they're not sure wether it is due to its modularity (which they define as &quot;many clusters (modules) of highly connected neurons that are only sparsely connected to neurons in other modules&quot;), or its sparsity (or probably a combination of both):</p>
  <blockquote>
  <p>However, it cannot be ruled out that sparsity itself mitigates catastrophic forgetting <a href="https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004128" rel="nofollow noreferrer">1</a>, or that the general learning abilities of the network have been improved due to the separation into a skill module and a learning module. Either way, the data support our hypothesis that a connection cost promotes the evolution of sparsity, modularity, and increased performance on learning tasks.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-30T12:52:50.507Z
Answers.LastActivityDate: 2024-05-30T12:52:50.507Z
Answers.UserDisplayName: David
Body: <p>Having recurrency is intuitevely linked with modularity for me (you would need a network topology with an infinite number of layers to account for every combinaisons of situtations possible), but it's not clear for me wether recurrent sparse networks are more modular than recurrent ones with continuous activations.
  Are brains very sparse and modular only out of energy consideration or is it required to their composition/blending of concepts, etc.?</p>
  <p>Edit: Modified my question according to my comment</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-30T11:37:26.133Z
Comments.Text: this is mere speculation imho
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-05-30T12:03:48.530Z
Comments.Text: Wikipedia says that "networks with high modularity have dense connections between the nodes within modules but sparse connections between nodes in different modules". If correct, I should reformulate it as _exhibit_ the same behaviour as modular network (e.g. combinaison to form new knowledge, resilience to catastrophic forgeting).
Comments.UserDisplayName: David
Tags: <neural-networks><recurrent-neural-networks><topology>
Title: Is sparsity required for behaviour exhibited by modular neural networks such as compositional generalisation, resiliency to catastrophic learning?
UserDisplayName: David
ThreadText: # Is sparsity required for behaviour exhibited by modular neural networks such as compositional generalisation, resiliency to catastrophic learning?
  
  (Asked by: username_0 on 2024-05-29)
  
  Having recurrency is intuitevely linked with modularity for me (you would need a network topology with an infinite number of layers to account for every combinaisons of situtations possible), but it's not clear for me wether recurrent sparse networks are more modular than recurrent ones with continuous activations.
  Are brains very sparse and modular only out of energy consideration or is it required to their composition/blending of concepts, etc.?
  
  Edit: Modified my question according to my comment
  
  **Tags:** neural-networks, recurrent-neural-networks, topology
  
  > **Comments:**
  > **username_1**: this is mere speculation imho  
  > **username_0**: Wikipedia says that "networks with high modularity have dense connections between the nodes within modules but sparse connections between nodes in different modules". If correct, I should reformulate it as _exhibit_ the same behaviour as modular network (e.g. combinaison to form new knowledge, resilience to catastrophic forgeting).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2024-05-30. Score: 0 | Accepted answer
  
  According to [research I found](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004128), they show that it helps the network generalise and avoid catastrophic forgeting, but they're not sure wether it is due to its modularity (which they define as "many clusters (modules) of highly connected neurons that are only sparsely connected to neurons in other modules"), or its sparsity (or probably a combination of both):
  
  > However, it cannot be ruled out that sparsity itself mitigates catastrophic forgetting [1](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004128), or that the general learning abilities of the network have been improved due to the separation into a skill module and a learning module. Either way, the data support our hypothesis that a connection cost promotes the evolution of sparsity, modularity, and increased performance on learning tasks.
  
  
  ---
  
  