Answers.Body: <p>There is indeed an investigation in progress, regarding this topic. A first publication from last march noted that modularity has been done, although not explicitly, since some time ago, but somehow training keeps being monolithic. This paper assess some primary questions about the matter and compares training times and performances on modular and heavily recurrent neural networks. See:</p>
  <ul>
  <li><a href="http://ceur-ws.org/Vol-2350/paper8.pdf" rel="nofollow noreferrer">Castillo-Bolado et al. Modularity as a Means for Complexity Management in Neural Networks Learning</a></li>
  </ul>
  <p>Some others are very focused on modularity, but staying with the monolithic training (see <a href="https://people.eecs.berkeley.edu/%7Ejda/pubs.html" rel="nofollow noreferrer">Jacob Andrea's research</a>, specially <em>Learning to reason</em> is very related to your third question). Somewhere between late 2019 and march next year, there should be more results (edit: <a href="https://www.sciencedirect.com/science/article/pii/S0893608021001222?via%3Dihub" rel="nofollow noreferrer">mentioned results here</a>).</p>
  <p>In relation to your two last questions, we're starting to see now that modularity is a major key towards generalisation. Let me recommend you some papers (you can find them all in arxiv or google scholar):</p>
  <ul>
  <li><p>Stochastic Adaptive Neural Architecture Search for Keyword Spotting (variations of an architecture to balance performance and resource usage).</p>
  </li>
  <li><p>Making Neural Programming Architectures Generalize via Recursion (they do task submodularity and I believe it's the first time that generalisation is guaranteed within field of neural networks).</p>
  </li>
  <li><p>Mastering the game of Go with deep neural networks and tree search (network topology is actually the search tree itself, you can see more of this if you look for graph neural networks).</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-06T12:24:57.623Z
Answers.LastActivityDate: 2022-02-10T10:08:54.937Z
Answers.LastEditDate: 2022-02-10T10:08:54.937Z
Answers.UserDisplayName: David
Answers.Body: <p>A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.<sup>1</sup> We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. Modularization as systems grow in complexity and demands on performance and capability grow is as old as industrialization.</p>
  
  <p>Our laboratory works with robotic control, thermodynamic instrumentation, and data analysis, artificial networks are components in these larger system contexts. we have no single MLPs or RNNs that by themselves perform any useful function.</p>
  
  <p>Contrary to the conjecture about hierarchies decades ago, the topology approach that seems to work well in most cases follows the more common system module relationships that are seen in power plants, automated factories, aeronautics, enterprise information architectures, and other complex engineered creations. The connections are those of flow, and if those are designed well, oversight functions are minimal. Flow occurs between modules involving protocols for communications, and each module performs its function well, encapsulating the lower level of complexity and functional detail. It is not one network overseeing another that seems to emerge most effective in actual practice but balance and symbiosis. Identification of clear master-slave design in the human brain seems to be equally slippery. It may be naive to assume that some all knowing subsystem can be a single causal factor in the remarkable collaborative synchronization of a number of other subsystems.</p>
  
  <p>The challenge is not finding the information paths that make up the system information topology. The information flow is often obvious upon problem analysis. The difficulty is in discovering the best of strategies to train these independent networks. Training dependencies are common and often critical, whereas in animals, the training occurs <em>in situ</em> or not at all. We're discovering conditions under which that kind of learning in our systems is practical and how to achieve it. Most of our research along these lines is intended to discover ways to achieve higher reliability and lower burden in terms of research hours to get it.</p>
  
  <p>Higher functionality is not always of benefit. It often produces lower reliability and consumes additional development resources with little return. Find a way to marry higher level automation, resource thrift, and reliability into one development process, and you might win an award and honorable mention around the web.</p>
  
  <p>Parallel systems that have the same objective is a good idea, but not a new one. In one aeronautics system, nine parallel systems have the same objective, in groups of three. Each group uses a different computing approach. If two of the systems using the same approach provide the same output and the third differs, the matching output is used and the difference in the third is reported as a system fault. If two of the different approaches provide similar results and the third differs substantially, a merge of the two similar results is used and the third is reported as a use case to further develop the dissenting approach.</p>
  
  <p>The improved fault tolerance has a cost, eight more systems and associated computing resources and connectivity plus the comparators at the tail, but in systems that are a matter of life and death, the extra costs are paid and the reliability is maximized.</p>
  
  <p>Dynamic topological adaptation is related to redundant systems and fault tolerance but in some ways is quite distinct. In that area of development, the technology to follow is neuromorphic computing, which is partly inspired by neuroplasticity.</p>
  
  <p>One last distinction to consider is between process topology, data topology, and hardware topology. These three geometric frames can produce greater efficiency together if aligned in specific ways that produce more direct mappings between the relationships between flow, representation, and mechanics. They are, however, distinct topologies. The meaning of alignment may not be apparent without diving deeply into these concepts and the details that emerge for specific product or service objectives.</p>
  
  <p><strong>Footnotes</strong></p>
  
  <p>[1] Deep networks that are trained as a single unit and function without connectivity to other artificial networks are not necessarily monolithic. Most practical deep networks have heterogeneous sequence of layers in terms of their activation functions and often their cell types.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-05T06:42:46.440Z
Answers.LastActivityDate: 2018-12-12T11:28:04.203Z
Answers.LastEditDate: 2018-12-12T11:28:04.203Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>Modular/Multiple Neural networks (MNNs) revolve around training smaller, independent networks that can feed into each other or another higher network.</p>
  
  <p>In principle, the hierarchical organization could allow us to make sense of more complex problem spaces and reach a higher functionality, but it seems difficult to find examples of concrete research done in the past regarding this. I've found a few sources:</p>
  
  <p><a href="https://en.wikipedia.org/wiki/Modular_neural_network" rel="noreferrer">https://en.wikipedia.org/wiki/Modular_neural_network</a></p>
  
  <p><a href="https://www.teco.edu/~albrecht/neuro/html/node32.html" rel="noreferrer">https://www.teco.edu/~albrecht/neuro/html/node32.html</a></p>
  
  <p><a href="https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&amp;isAllowed=y" rel="noreferrer">https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&amp;isAllowed=y</a> </p>
  
  <p>A few concrete questions I have:</p>
  
  <ul>
  <li><p><em>Has there been any recent research into the use of MNNs?</em></p></li>
  <li><p><em>Are there any tasks where MNNs have shown better performance than large single nets?</em></p></li>
  <li><p><em>Could MNNs be used for multimodal classification, i.e. train each net on a fundamentally different type of data, (text vs image) and feed forward to a higher level intermediary that operates on all the outputs?</em></p></li>
  <li><p><em>From a software engineering perspective, aren't these more fault tolerant and easily isolatable on a distributed system?</em></p></li>
  <li><p><em>Has there been any work into dynamically adapting the topologies of subnetworks using a process like Neural Architecture Search?</em></p></li>
  <li><p><em>Generally, are MNNs practical in any way?</em></p></li>
  </ul>
  
  <p>Apologies if these questions seem naive, I've just come into ML and more broadly CS from a biology/neuroscience background and am captivated by the potential interplay. </p>
  
  <p>I really appreciate you taking the time and lending your insight!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-05-12T17:00:45.543Z
Comments.Text: I'd thought about implementation of such a modular system to improve performance, and filter out - *definitely* - unnecessary dependencies on inputs. Thought it was a eureka moment, but didn't know it was already an established structure.
Comments.UserDisplayName: Tobi
Tags: <neural-networks><topology><architecture><neurons><biology>
Title: Are Modular Neural Networks more effective than large, monolithic networks at any tasks?
UserDisplayName: Harsh Sikka
ThreadText: # Are Modular Neural Networks more effective than large, monolithic networks at any tasks?
  
  (Asked by: username_0 on 2018-12-02)
  
  Modular/Multiple Neural networks (MNNs) revolve around training smaller, independent networks that can feed into each other or another higher network.
  
  In principle, the hierarchical organization could allow us to make sense of more complex problem spaces and reach a higher functionality, but it seems difficult to find examples of concrete research done in the past regarding this. I've found a few sources:
  
  <https://en.wikipedia.org/wiki/Modular_neural_network>
  
  <https://www.teco.edu/~albrecht/neuro/html/node32.html>
  
  <https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&isAllowed=y>
  
  A few concrete questions I have:
  
  * *Has there been any recent research into the use of MNNs?*
  * *Are there any tasks where MNNs have shown better performance than large single nets?*
  * *Could MNNs be used for multimodal classification, i.e. train each net on a fundamentally different type of data, (text vs image) and feed forward to a higher level intermediary that operates on all the outputs?*
  * *From a software engineering perspective, aren't these more fault tolerant and easily isolatable on a distributed system?*
  * *Has there been any work into dynamically adapting the topologies of subnetworks using a process like Neural Architecture Search?*
  * *Generally, are MNNs practical in any way?*
  
  Apologies if these questions seem naive, I've just come into ML and more broadly CS from a biology/neuroscience background and am captivated by the potential interplay.
  
  I really appreciate you taking the time and lending your insight!
  
  **Tags:** neural-networks, topology, architecture, neurons, biology
  > **Comments:**
  > **username_1**: I'd thought about implementation of such a modular system to improve performance, and filter out - *definitely* - unnecessary dependencies on inputs. Thought it was a eureka moment, but didn't know it was already an established structure.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-08-06. Score: 3
  
  There is indeed an investigation in progress, regarding this topic. A first publication from last march noted that modularity has been done, although not explicitly, since some time ago, but somehow training keeps being monolithic. This paper assess some primary questions about the matter and compares training times and performances on modular and heavily recurrent neural networks. See:
  
  * [Castillo-Bolado et al. Modularity as a Means for Complexity Management in Neural Networks Learning](http://ceur-ws.org/Vol-2350/paper8.pdf)
  
  Some others are very focused on modularity, but staying with the monolithic training (see [Jacob Andrea's research](https://people.eecs.berkeley.edu/%7Ejda/pubs.html), specially *Learning to reason* is very related to your third question). Somewhere between late 2019 and march next year, there should be more results (edit: [mentioned results here](https://www.sciencedirect.com/science/article/pii/S0893608021001222?via%3Dihub)).
  
  In relation to your two last questions, we're starting to see now that modularity is a major key towards generalisation. Let me recommend you some papers (you can find them all in arxiv or google scholar):
  
  * Stochastic Adaptive Neural Architecture Search for Keyword Spotting (variations of an architecture to balance performance and resource usage).
  * Making Neural Programming Architectures Generalize via Recursion (they do task submodularity and I believe it's the first time that generalisation is guaranteed within field of neural networks).
  * Mastering the game of Go with deep neural networks and tree search (network topology is actually the search tree itself, you can see more of this if you look for graph neural networks).
  
  
  ---
  
  ### Answer by username_3 on 2018-12-05. Score: 2
  
  A benchmark comparison of systems comprised of separately trained networks relative to single deeper networks would not likely reveal a universally applicable best choice.1 We can see in the literature the increase in the number of larger systems where several artificial networks are combined, along with other types of components. It is to be expected. Modularization as systems grow in complexity and demands on performance and capability grow is as old as industrialization.
  
  Our laboratory works with robotic control, thermodynamic instrumentation, and data analysis, artificial networks are components in these larger system contexts. we have no single MLPs or RNNs that by themselves perform any useful function.
  
  Contrary to the conjecture about hierarchies decades ago, the topology approach that seems to work well in most cases follows the more common system module relationships that are seen in power plants, automated factories, aeronautics, enterprise information architectures, and other complex engineered creations. The connections are those of flow, and if those are designed well, oversight functions are minimal. Flow occurs between modules involving protocols for communications, and each module performs its function well, encapsulating the lower level of complexity and functional detail. It is not one network overseeing another that seems to emerge most effective in actual practice but balance and symbiosis. Identification of clear master-slave design in the human brain seems to be equally slippery. It may be naive to assume that some all knowing subsystem can be a single causal factor in the remarkable collaborative synchronization of a number of other subsystems.
  
  The challenge is not finding the information paths that make up the system information topology. The information flow is often obvious upon problem analysis. The difficulty is in discovering the best of strategies to train these independent networks. Training dependencies are common and often critical, whereas in animals, the training occurs *in situ* or not at all. We're discovering conditions under which that kind of learning in our systems is practical and how to achieve it. Most of our research along these lines is intended to discover ways to achieve higher reliability and lower burden in terms of research hours to get it.
  
  Higher functionality is not always of benefit. It often produces lower reliability and consumes additional development resources with little return. Find a way to marry higher level automation, resource thrift, and reliability into one development process, and you might win an award and honorable mention around the web.
  
  Parallel systems that have the same objective is a good idea, but not a new one. In one aeronautics system, nine parallel systems have the same objective, in groups of three. Each group uses a different computing approach. If two of the systems using the same approach provide the same output and the third differs, the matching output is used and the difference in the third is reported as a system fault. If two of the different approaches provide similar results and the third differs substantially, a merge of the two similar results is used and the third is reported as a use case to further develop the dissenting approach.
  
  The improved fault tolerance has a cost, eight more systems and associated computing resources and connectivity plus the comparators at the tail, but in systems that are a matter of life and death, the extra costs are paid and the reliability is maximized.
  
  Dynamic topological adaptation is related to redundant systems and fault tolerance but in some ways is quite distinct. In that area of development, the technology to follow is neuromorphic computing, which is partly inspired by neuroplasticity.
  
  One last distinction to consider is between process topology, data topology, and hardware topology. These three geometric frames can produce greater efficiency together if aligned in specific ways that produce more direct mappings between the relationships between flow, representation, and mechanics. They are, however, distinct topologies. The meaning of alignment may not be apparent without diving deeply into these concepts and the details that emerge for specific product or service objectives.
  
  **Footnotes**
  
  [1] Deep networks that are trained as a single unit and function without connectivity to other artificial networks are not necessarily monolithic. Most practical deep networks have heterogeneous sequence of layers in terms of their activation functions and often their cell types.
  
  
  ---
  
  