Answers.Body: <p><strong>Solution</strong></p>
  
  <p>In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.</p>
  
  <p>The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.</p>
  
  <ul>
  <li>Encoding of ordinals</li>
  <li>Lossless conversion of IEEE floats</li>
  <li>Serialization of the structure</li>
  <li>Lossless compression of the stream</li>
  </ul>
  
  <p>Upon retrieval, the elements are reversed.</p>
  
  <ul>
  <li>Decompression of the stream</li>
  <li>Parsing to reassemble structure</li>
  <li>Expansion of IEEE float representations</li>
  <li>Decoding of ordinals</li>
  </ul>
  
  <p>The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.</p>
  
  <p>If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.</p>
  
  <p><strong>Notions of General Intelligence</strong></p>
  
  <p>Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.</p>
  
  <ul>
  <li>The complexity of networks in the brain</li>
  <li>That there is a significant varieties of neurons in brains</li>
  <li>The impact of neuroplasticity</li>
  <li>Complexity of chemical signaling independent from nerves</li>
  <li>Whether intelligence is in fact multi-dimensional</li>
  <li>Large number of genes that correlate with intelligence</li>
  <li>The academic slant of concepts regarding intelligence</li>
  <li>Difference between brain signal topology and computing models </li>
  <li>Data-centricity</li>
  </ul>
  
  <p>There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.</p>
  
  <ul>
  <li><a href="https://ai.stackexchange.com/questions/9405/why-did-artificial-intelligence-projects-fail">Why did Artificial Intelligence projects fail?</a></li>
  <li><a href="https://ai.stackexchange.com/questions/41/can-the-iq-of-an-ai-program-be-measured">Can the IQ of an AI program be measured?</a></li>
  <li><a href="https://ai.stackexchange.com/questions/8219/can-a-brain-be-intelligent-without-a-body">Can a brain be intelligent without a body?</a></li>
  <li><a href="https://ai.stackexchange.com/questions/7337/is-the-singularity-concept-mathematically-flawed">Is the singularity concept mathematically flawed?</a></li>
  <li><a href="https://ai.stackexchange.com/questions/4987/is-there-a-ceiling-to-intelligence-itself/5045#5045">Is there a ceiling to intelligence itself?</a></li>
  <li><a href="https://ai.stackexchange.com/questions/7573/self-conscious-artifical-intelligence/7576#7576">Self-conscious artifical intelligence</a></li>
  <li><a href="https://ai.stackexchange.com/questions/15/is-the-turing-test-or-any-of-its-variants-a-reliable-test-of-artificial-intell">Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?</a></li>
  </ul>
  
  <p>There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-31T10:12:04.577Z
Answers.Text: I am well aware of the caveats concerning the construct validity of intelligence, the nature of consciousness, and neurophysiological realism. I am currently focusing on aspects of implementation. What is possible with current software engineering and ML knowledge? For example, in your answer to "Why did [AI] projects fail?" you mention the importance of parallel over serial processing. However, when I read open code that solves specific problems I am concerned that those solutions are too slow and not suitable for AGI. There is a bias toward exact solutions and less emphasis on heuristics.
Answers.UserDisplayName: noumenal
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-12-28T00:18:30.347Z
Answers.LastActivityDate: 2018-12-28T00:18:30.347Z
Answers.UserDisplayName: Douglas Daseeco
Body: <p>I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using <em>sparse coding</em> <sup>[1]</sup>. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.</p>
  
  <p>A default cognitive model of short-term memory can be separated in three different stages:</p>
  
  <blockquote>Encoding → Maintenance → Retrieval
  </blockquote>
  
  <p>One solution is to use specialized hardware <sup>[2][3]</sup>, but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.</p>
  
  <p>With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).</p>
  
  <ul>
  <li>How can a specific <em>connectivity pattern</em> be stored in an <em>optimally
  compact</em> representation?</li>
  <li>Is there an implementation with concrete example code?</li>
  <li>What is the state-of-the-art?<sup>*</sup></li>
  </ul>
  
  <p><sub>* I will not specify "real-time" here, but the context is humanoid AGI. </sub></p>
  
  <hr>
  
  <p><sub> 
  [1] Papyan, V., Romano, Y., &amp; Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. <em>Journal on Machine Learning Research, 18</em>(83): 1–52. arXiv:1607.08194
  <a href="http://jmlr.org/papers/volume18/16-505/16-505.pdf" rel="nofollow noreferrer">http://jmlr.org/papers/volume18/16-505/16-505.pdf</a>
  </sub></p>
  
  <p><sub>[2] LeGallo et al. (2018). Mixed-precision in-memory computing. <a href="https://www.nature.com/articles/s41928-018-0054-8" rel="nofollow noreferrer">https://www.nature.com/articles/s41928-018-0054-8</a>
  </sub></p>
  
  <p><sub>
  [3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. <a href="https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/" rel="nofollow noreferrer">https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/</a></sub></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T01:02:37.230Z
Comments.Text: Welcome to SE:AI!  I assume this is a hypothetical AGI we are discussing, but I feel like this is a deep encoding question involving binary representation.  (Continuous learning also seems ancillary, but is good context for the problem:) I tweaked the title to reflect the central question.
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-12-20T12:01:17.517Z
Comments.Text: Thanks @DukeZhou! It's actually some theoretical groundwork for a basic AGI that I wish to implement IRL. Unfortunately, "deep encoding" doesn't seem to be a popular keyword. Thanks for pointing me to "Continuous learning". I think information-theory is applicable, but requires two Shannon-streams as the input *destination* is the temporary memory storage. Just pointing me to the right keywords is very helpful for my research.
Comments.UserDisplayName: noumenal
Tags: <optimization><agi><memory>
Title: How can a specific connectivity pattern be stored in an optimally compact representation?
UserDisplayName: noumenal
ThreadText: # How can a specific connectivity pattern be stored in an optimally compact representation?
  
  (Asked by: username_0 on 2018-12-19)
  
  I am interested in optimizing the memory capacity of an AGI. Given a specific complex input an AI can create a simplified model. This is a problem that can be solved using *sparse coding* [1]. However, this solves only the problem of encoding and not the maintenance of online representations—in cognitive terms: the state of mind.
  
  A default cognitive model of short-term memory can be separated in three different stages:
  
  > Encoding → Maintenance → Retrieval
  
  One solution is to use specialized hardware [2][3], but I am interested in software approaches to this problem and I would thus like to emphasize that it is the digital representation, which I am most interested in.
  
  With the exception of qubits, the smallest possible representation are binary digits. However, additional architecture is required to represent phase spaces (i.e. floating point precision memory) and higher-order representations maybe include arrays and dictionaries. (Optimizing these are trivial... or simply to be postponed until needed, according to Knuth).
  
  * How can a specific *connectivity pattern* be stored in an *optimally
    compact* representation?
  * Is there an implementation with concrete example code?
  * What is the state-of-the-art?\*
  
  \* I will not specify "real-time" here, but the context is humanoid AGI.
  
  ---
  
  [1] Papyan, V., Romano, Y., & Elad, M. (2017). Convolutional Neural Networks Analyzed via Convolutional Sparse Coding. *Journal on Machine Learning Research, 18*(83): 1–52. arXiv:1607.08194
  <http://jmlr.org/papers/volume18/16-505/16-505.pdf>
  
  [2] LeGallo et al. (2018). Mixed-precision in-memory computing. <https://www.nature.com/articles/s41928-018-0054-8>
  
  [3] IBM. (2018). IBM Scientists Demonstrate Mixed-Precision In-Memory Computing for the First Time; Hybrid Design for AI Hardware. <https://www.ibm.com/blogs/research/2018/04/ibm-scientists-demonstrate-mixed-precision-in-memory-computing-for-the-first-time-hybrid-design-for-ai-hardware/>
  
  **Tags:** optimization, agi, memory
  
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  I assume this is a hypothetical AGI we are discussing, but I feel like this is a deep encoding question involving binary representation.  (Continuous learning also seems ancillary, but is good context for the problem:) I tweaked the title to reflect the central question.  
  > **username_0**: Thanks @username_1! It's actually some theoretical groundwork for a basic AGI that I wish to implement IRL. Unfortunately, "deep encoding" doesn't seem to be a popular keyword. Thanks for pointing me to "Continuous learning". I think information-theory is applicable, but requires two Shannon-streams as the input *destination* is the temporary memory storage. Just pointing me to the right keywords is very helpful for my research.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-12-28. Score: 2
  
  **Solution**
  
  In learning systems, the output of learning is a set of parameters. These parameters can be a tensor in the case of a CNN, a representation of a directed graph in the case of recursive network designs, or any other structure holding the results of training.
  
  The general case of compactly storing the result of training involves four process elements. These elements can be combined into one algorithm if needed.
  
  * Encoding of ordinals
  * Lossless conversion of IEEE floats
  * Serialization of the structure
  * Lossless compression of the stream
  
  Upon retrieval, the elements are reversed.
  
  * Decompression of the stream
  * Parsing to reassemble structure
  * Expansion of IEEE float representations
  * Decoding of ordinals
  
  The encoding of ordinals requires the mapping of values to the minimal number of bytes or, most optimally, bits. The mapping must be reversible, meaning that there can be no information loss, so that decoding is completely reliable.
  
  If the encoding and serialization packs the information tightly, there may be little gain with a general compression algorithm such as LZ4.
  
  **Notions of General Intelligence**
  
  Before devoting much time on developing humanoid AGI and modeling cognition and states of mind, it may be useful to curb expectations away from trendy ideas. There is much conflict between the naive conceptions of general intelligence and the realities of neurological systems in animals and people, which are far more sophisticated than implied in the trendy media hype. The most important considerations are these.
  
  * The complexity of networks in the brain
  * That there is a significant varieties of neurons in brains
  * The impact of neuroplasticity
  * Complexity of chemical signaling independent from nerves
  * Whether intelligence is in fact multi-dimensional
  * Large number of genes that correlate with intelligence
  * The academic slant of concepts regarding intelligence
  * Difference between brain signal topology and computing models
  * Data-centricity
  
  There are many articles on the web on these topics, along with critiques of concepts such as the g-factor and the idea of general problem solving and general intelligence. There are several questions here on this topic.
  
  * [Why did Artificial Intelligence projects fail?](https://ai.stackexchange.com/questions/9405/why-did-artificial-intelligence-projects-fail)
  * [Can the IQ of an AI program be measured?](https://ai.stackexchange.com/questions/41/can-the-iq-of-an-ai-program-be-measured)
  * [Can a brain be intelligent without a body?](https://ai.stackexchange.com/questions/8219/can-a-brain-be-intelligent-without-a-body)
  * [Is the singularity concept mathematically flawed?](https://ai.stackexchange.com/questions/7337/is-the-singularity-concept-mathematically-flawed)
  * [Is there a ceiling to intelligence itself?](https://ai.stackexchange.com/questions/4987/is-there-a-ceiling-to-intelligence-itself/5045#5045)
  * [Self-conscious artifical intelligence](https://ai.stackexchange.com/questions/7573/self-conscious-artifical-intelligence/7576#7576)
  * [Is the Turing Test, or any of its variants, a reliable test of artificial intelligence?](https://ai.stackexchange.com/questions/15/is-the-turing-test-or-any-of-its-variants-a-reliable-test-of-artificial-intell)
  
  There is no value in discouraging curiosity and serious research into understanding the nature of intelligence or efforts to expanding the boundaries of computing. The above comments are to assist in bringing the starting point of such inquiry into the realm of enthusiasm anchored by scientific methodology. Conjecture makes for great movie plots, but does not further science and does not lead to real world technology products and services.
  
  > **Comments:**
  > **username_0**: I am well aware of the caveats concerning the construct validity of intelligence, the nature of consciousness, and neurophysiological realism. I am currently focusing on aspects of implementation. What is possible with current software engineering and ML knowledge? For example, in your answer to "Why did [AI] projects fail?" you mention the importance of parallel over serial processing. However, when I read open code that solves specific problems I am concerned that those solutions are too slow and not suitable for AGI. There is a bias toward exact solutions and less emphasis on heuristics.  
  
  ---
  
  