Answers.Body: <p>Indeed the theoretical recursive isomorphism of knowledge representation formalisms (KRFs) may overstate the theoretical equivalence, overlooking practical differences in efficiency, scalability, and applicability. Thus while the claim is intriguing, it may not adequately address the strengths of the dominant connectionist methods in AI especially for fast pattern recognition in perceptual applications. Each formalism has their own <a href="https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence" rel="nofollow noreferrer">applicability</a>.</p>
  <blockquote>
  <p>With the rise of deep learning, the symbolic AI approach has been compared to deep learning as complementary &quot;...with parallels having been drawn many times by AI researchers between Kahneman's research on human reasoning and decision making – reflected in his book Thinking, Fast and Slow – and the so-called &quot;AI systems 1 and 2&quot;, which would in principle be modelled by deep learning and symbolic reasoning, respectively.&quot; In this view, symbolic reasoning is more apt for deliberative reasoning, planning, and explanation while deep learning is more apt for fast pattern recognition in perceptual applications with noisy data.</p>
  </blockquote>
  <p>In fact there's significant effort from symbolic AI's KRF to try to build a large KB about the world, focusing on implicit knowledge such as <a href="https://en.wikipedia.org/wiki/Cyc" rel="nofollow noreferrer">Cyc</a>. It's an example of limited practical applicability and scalability of the pure symbolic AI formalism.</p>
  <blockquote>
  <p>Cyc is a long-term artificial intelligence project that aims to assemble a comprehensive ontology and knowledge base that spans the basic concepts and rules about how the world works. Hoping to capture common sense knowledge, Cyc focuses on implicit knowledge... Cyc's ontology grew to about 100,000 terms in 1994, and as of 2017, it contained about 1,500,000 terms. The Cyc knowledge base involving ontological terms was largely created by hand axiom-writing; it was at about 1 million in 1994, and as of 2017, it is at about 24.5 million. In 2008, Cyc resources were mapped to many Wikipedia articles... The Cyc inference engine performs general logical deduction. It also performs inductive reasoning, statistical machine learning and symbolic machine learning, and abductive reasoning...  Machine-learning scientist Pedro Domingos refers to the project as a &quot;catastrophic failure&quot; for the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T01:18:15.827Z
Answers.Text: In the conclusion of the paper, the authors argue that "the existence of recursive isomorphisms between KRFs implies that for any knowledge operator (e.g., gradient descent) in a KRF we can effectively find an operator in another (isomorphic) KRF to perform the same transformation." So, in this sense, are efficiency, scalability, and applicability truly inherent advantages of neural networks?
Answers.UserDisplayName: Alan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T01:43:08.813Z
Answers.Text: Since SGD+minibatch can train large stacked NNs such as transformers effectively without overfitting, NNs do have an advantage for training efficiency and scalability. Note the authors in your paper mentions the condition for recursive isomorphism: *up to an offline compilation*.  In theory this condition seems not a big concern, but as Cyc shown in my answer, this could have a big practical scalability issue if it wants to match latest LLMs' common sense and implicit knowledge representation and generalization, since all those 'hand axiom-writings' are offline compilation. Hope it clarifies.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T02:19:29.190Z
Answers.Text: Thank you very much for your response! So, my understanding is that whether a KRF that is recursively isomorphic to neural networks possesses good efficiency, scalability, and applicability depends on the computational efficiency of the aforementioned recursive isomorphism (i.e., the offline compilation). For instance, if this recursive isomorphism can be computed in linear time, then the relevant techniques of neural networks can be effectively converted to the target KRF.
Answers.UserDisplayName: Alan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T03:16:51.437Z
Answers.Text: You're on the right track. Intuitively it's not surprising for such isomorphism since anything can be programmed in declarative functional programming languages can be achieved in imperative procedural programming languages so long as they're Turing-complete, albeit some cases one type language can be much more efficient. Similarly there's a famous theoretical Curry-Howard Isomorphism between declarative constructive logic and procedural computation. The lesson so far is that everyday world common sense and implicit knowledge belongs to thinking-fast pattern recognition tasks including LLMs.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T03:25:54.467Z
Answers.Text: The real challenge here to transform SGD to its corresponding method in Cyc like symbolic AI KRF, so far we haven't found an efficient scalable self-evolving method without those offline compilation of hand axiom-writings for each microtheory within the whole world ontology. For example, if new data fact from a database gives a new type of fruit, the symbolic KRF's fruitFn() function has to be modified and recompiled which is clearly an overfitting case in the sense of statistical inference, thus is not generalizable compared to LLMs KRF.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T04:14:04.367Z
Answers.Text: I am uncertain whether the Curry-Howard Isomorphism plays the same role. It appears that this isomorphism concerns the relationship between proofs and computation, whereas the recursive isomorphism discussed in the aforementioned work, when placed in a similar context, pertains to the relationship between declarative and procedural representations of knowledge. These do not seem to be the same thing.
Answers.UserDisplayName: Alan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T04:49:48.473Z
Answers.Text: Constructive logic + type systems such as Coq/Agda does represent knowledge in a Turing-complete declarative way and most logics are inherently explicit declarative representation of knowledge, thus most logics are hard to capture implicit knowledge which can be better handled by connectionist procedural functional models. For example after seeing a cat and a dog which are explicit declarative knowledge representations it's hard for logic to infer an animal which is in between. Thus the recursive isomorphism in OP could be viewed as the type system isomorphic to NN KRF which approximates TM.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-24T06:26:06.807Z
Answers.Text: I understand your point, but such an analogy still seems somewhat superficial. Whether the two kinds of isomorphisms are indeed the same requires a more rigorous argument. Anyway, thank you for your response:)
Answers.UserDisplayName: Alan
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-12-23T07:41:30.540Z
Answers.LastActivityDate: 2024-12-23T07:41:30.540Z
Answers.UserDisplayName: cinch
Body: <p>In a recent arXiv paper (accepted by AAAI-25):</p>
  <p><a href="https://arxiv.org/abs/2412.11855" rel="nofollow noreferrer">https://arxiv.org/abs/2412.11855</a>
  (Title: A Theory of Formalisms for Representing Knowledge)</p>
  <p>the authors present a seemingly intriguing result, claiming that all universal (or natural and equally expressive) knowledge representation formalisms are recursively isomorphic. To establish this result, the authors developed a framework to capture the class of possible knowledge representation formalisms and define what constitutes a universal knowledge representation formalism.</p>
  <p>I am wondering if this work has any limitations. More specifically:</p>
  <ol>
  <li><p>Is the framework proposed for knowledge representation formalisms defined in a reasonable and comprehensive manner?</p>
  </li>
  <li><p>Is the claim of recursive isomorphisms among universal (or natural and equally expressive) knowledge representation formalisms correct? If so, what are its implications for AI?</p>
  </li>
  <li><p>In the conclusion of this paper, the authors claim:</p>
  </li>
  </ol>
  <blockquote>
  <p>For the debate between symbolic AI and connectionist AI, the existence of recursive isomorphisms between knowledge representation formalisms (KRFs) implies that for any knowledge operator (e.g., gradient descent) in one KRF, we can effectively find an operator in another (isomorphic) KRF to perform the same transformation. From a theoretical perspective, all these representation methodologies either pave the way to AGI or none, with core challenges being universal and advancements in one methodology benefiting others.</p>
  </blockquote>
  <p>However, it appears that knowledge representation formalisms based on neural networks have achieved a dominant position in contemporary AI. Therefore, does the above claim potentially overstate its significance?</p>
  
Tags: <neural-networks><logic><knowledge-representation><symbolic-ai><knowledge-based-systems>
Title: Are all (natural) knowledge representation formalisms created equal?
UserDisplayName: Alan
ThreadText: # Are all (natural) knowledge representation formalisms created equal?
  
  (Asked by: username_0 on 2024-12-23)
  
  In a recent arXiv paper (accepted by AAAI-25):
  
  <https://arxiv.org/abs/2412.11855>
  (Title: A Theory of Formalisms for Representing Knowledge)
  
  the authors present a seemingly intriguing result, claiming that all universal (or natural and equally expressive) knowledge representation formalisms are recursively isomorphic. To establish this result, the authors developed a framework to capture the class of possible knowledge representation formalisms and define what constitutes a universal knowledge representation formalism.
  
  I am wondering if this work has any limitations. More specifically:
  
  1. Is the framework proposed for knowledge representation formalisms defined in a reasonable and comprehensive manner?
  2. Is the claim of recursive isomorphisms among universal (or natural and equally expressive) knowledge representation formalisms correct? If so, what are its implications for AI?
  3. In the conclusion of this paper, the authors claim:
  
  > For the debate between symbolic AI and connectionist AI, the existence of recursive isomorphisms between knowledge representation formalisms (KRFs) implies that for any knowledge operator (e.g., gradient descent) in one KRF, we can effectively find an operator in another (isomorphic) KRF to perform the same transformation. From a theoretical perspective, all these representation methodologies either pave the way to AGI or none, with core challenges being universal and advancements in one methodology benefiting others.
  
  However, it appears that knowledge representation formalisms based on neural networks have achieved a dominant position in contemporary AI. Therefore, does the above claim potentially overstate its significance?
  
  **Tags:** neural-networks, logic, knowledge-representation, symbolic-ai, knowledge-based-systems
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-12-23. Score: 1
  
  Indeed the theoretical recursive isomorphism of knowledge representation formalisms (KRFs) may overstate the theoretical equivalence, overlooking practical differences in efficiency, scalability, and applicability. Thus while the claim is intriguing, it may not adequately address the strengths of the dominant connectionist methods in AI especially for fast pattern recognition in perceptual applications. Each formalism has their own [applicability](https://en.wikipedia.org/wiki/Symbolic_artificial_intelligence).
  
  > With the rise of deep learning, the symbolic AI approach has been compared to deep learning as complementary "...with parallels having been drawn many times by AI researchers between Kahneman's research on human reasoning and decision making – reflected in his book Thinking, Fast and Slow – and the so-called "AI systems 1 and 2", which would in principle be modelled by deep learning and symbolic reasoning, respectively." In this view, symbolic reasoning is more apt for deliberative reasoning, planning, and explanation while deep learning is more apt for fast pattern recognition in perceptual applications with noisy data.
  
  In fact there's significant effort from symbolic AI's KRF to try to build a large KB about the world, focusing on implicit knowledge such as [Cyc](https://en.wikipedia.org/wiki/Cyc). It's an example of limited practical applicability and scalability of the pure symbolic AI formalism.
  
  > Cyc is a long-term artificial intelligence project that aims to assemble a comprehensive ontology and knowledge base that spans the basic concepts and rules about how the world works. Hoping to capture common sense knowledge, Cyc focuses on implicit knowledge... Cyc's ontology grew to about 100,000 terms in 1994, and as of 2017, it contained about 1,500,000 terms. The Cyc knowledge base involving ontological terms was largely created by hand axiom-writing; it was at about 1 million in 1994, and as of 2017, it is at about 24.5 million. In 2008, Cyc resources were mapped to many Wikipedia articles... The Cyc inference engine performs general logical deduction. It also performs inductive reasoning, statistical machine learning and symbolic machine learning, and abductive reasoning... Machine-learning scientist Pedro Domingos refers to the project as a "catastrophic failure" for the unending amount of data required to produce any viable results and the inability for Cyc to evolve on its own.
  
  > **Comments:**
  > **username_0**: In the conclusion of the paper, the authors argue that "the existence of recursive isomorphisms between KRFs implies that for any knowledge operator (e.g., gradient descent) in a KRF we can effectively find an operator in another (isomorphic) KRF to perform the same transformation." So, in this sense, are efficiency, scalability, and applicability truly inherent advantages of neural networks?  
  > **username_1**: Since SGD+minibatch can train large stacked NNs such as transformers effectively without overfitting, NNs do have an advantage for training efficiency and scalability. Note the authors in your paper mentions the condition for recursive isomorphism: *up to an offline compilation*.  In theory this condition seems not a big concern, but as Cyc shown in my answer, this could have a big practical scalability issue if it wants to match latest LLMs' common sense and implicit knowledge representation and generalization, since all those 'hand axiom-writings' are offline compilation. Hope it clarifies.  
  > **username_0**: Thank you very much for your response! So, my understanding is that whether a KRF that is recursively isomorphic to neural networks possesses good efficiency, scalability, and applicability depends on the computational efficiency of the aforementioned recursive isomorphism (i.e., the offline compilation). For instance, if this recursive isomorphism can be computed in linear time, then the relevant techniques of neural networks can be effectively converted to the target KRF.  
  > **username_1**: You're on the right track. Intuitively it's not surprising for such isomorphism since anything can be programmed in declarative functional programming languages can be achieved in imperative procedural programming languages so long as they're Turing-complete, albeit some cases one type language can be much more efficient. Similarly there's a famous theoretical Curry-Howard Isomorphism between declarative constructive logic and procedural computation. The lesson so far is that everyday world common sense and implicit knowledge belongs to thinking-fast pattern recognition tasks including LLMs.  
  > **username_1**: The real challenge here to transform SGD to its corresponding method in Cyc like symbolic AI KRF, so far we haven't found an efficient scalable self-evolving method without those offline compilation of hand axiom-writings for each microtheory within the whole world ontology. For example, if new data fact from a database gives a new type of fruit, the symbolic KRF's fruitFn() function has to be modified and recompiled which is clearly an overfitting case in the sense of statistical inference, thus is not generalizable compared to LLMs KRF.  
  > **username_0**: I am uncertain whether the Curry-Howard Isomorphism plays the same role. It appears that this isomorphism concerns the relationship between proofs and computation, whereas the recursive isomorphism discussed in the aforementioned work, when placed in a similar context, pertains to the relationship between declarative and procedural representations of knowledge. These do not seem to be the same thing.  
  > **username_1**: Constructive logic + type systems such as Coq/Agda does represent knowledge in a Turing-complete declarative way and most logics are inherently explicit declarative representation of knowledge, thus most logics are hard to capture implicit knowledge which can be better handled by connectionist procedural functional models. For example after seeing a cat and a dog which are explicit declarative knowledge representations it's hard for logic to infer an animal which is in between. Thus the recursive isomorphism in OP could be viewed as the type system isomorphic to NN KRF which approximates TM.  
  > **username_0**: I understand your point, but such an analogy still seems somewhat superficial. Whether the two kinds of isomorphisms are indeed the same requires a more rigorous argument. Anyway, thank you for your response:)  
  
  ---
  
  